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Overview 


This report presents the results of an innovative study designed to provide a more detailed under- 
standing of how parents and their teenage children were affected by the Opportunity NYC-Family 
Rewards program, a comprehensive conditional cash transfer program. The three-year program, 
launched by the Center for Economic Opportunity in the Mayor’s Office of the City of New York in 
2007, offered cash assistance to low-income families to reduce economic hardship. The cash 
incentives were tied to activities and outcomes in children’s education, family preventive health 
care, and parents’ employment, in the hopes of increasing families’ “human capital” and reducing 
their poverty in the long term. 

An evaluation by MDRC of the first two years of the Family Rewards program, published in 2010, 
found that it had positive effects on families’ economic well-being and mixed effects on children’s 
education, family health care, and parents’ employment. For example, while the program did not 
affect school outcomes for younger children, it substantially boosted the achievement of a subset of 
older children who were better prepared for high school when they entered the program in the ninth 
grade. Flow the program affected teenagers and their parents is the focus of this study, which has 
been embedded in MDRCs continuing core evaluation of the program. This report addresses key 
“pathways” that may underlie any effects of the program on teenagers (such as changes in the way 
teenagers spent their time) as well as outcomes that were not targeted but that the program may have 
affected (such as teenagers’ mental health, aggressive behavior, or substance use.) 

Key Findings 

Findings show that the Family Rewards program: 

• Changed how teenagers spent their time. For the subgroup of academically proficient teenagers, 
it increased the proportion of those who engaged primarily in academic activities and reduced 
the proportion who engaged primarily in social activities; 

• Increased parents’ spending on school-related and leisure expenses and increased the proportion 
of parents who saved for their children’s future education; 

• Flad no effects on parents’ monitoring of their teenage children’s activities or behavior and did 
not increase parent-teenager conflict or teenagers’ depression or anxiety; 

• Had no effects on teenagers’ sense of academic competence or their engagement in school but 
substantially reduced their problem behavior, such as aggression and substance use; 

• Did not reduce teenagers’ intrinsic motivation by paying them rewards for school attendance 
and academic achievement. 

This study of families with teenage children who participated in Family Rewards adds important 
information that will enhance understanding of the results of the core evaluation of the program. The 
next report on Family Rewards will examine the results after three years of the program; a final 
report will include two years of postprogram follow-up. 
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Preface 


What would happen if parents and their teenage children were offered cash incentives if the 
children went to school and passed their exams? Would teenagers spend more time on their 
schoolwork? Would offering incentives lead to tension and conflict between parents and 
children in an already challenging period in their lives? Would getting a reward for achievement 
in school reduce students’ intrinsic motivation to leam? 

Opportunity NYC-Family Rewards was a conditional cash transfer (CCT) program 
aimed to help families break the cycle of intergenerational poverty. CCT programs provide 
monetary incentives to families to take certain steps to invest in their own health, education, and 
economic potential. Inspired by similar initiatives in other countries, particularly Mexico’s 
Oportunidades program, Family Rewards was sponsored by New York City’s Center for 
Economic Opportunity in the Office of Mayor Michael R. Bloomberg and was delivered in six 
of New York City’s highest-poverty communities from 2007 to 2010. MDRC is evaluating the 
program’s effects using a randomized control trial research design. Our 2010 report on the 
program’s early findings found that it increased families’ economic well-being and had mixed 
effects — ranging from no effect to small positive effects — on children’s education, family 
health care, and parents’ work and training. Analysis of the longer-term impacts of the program 
is currently under way. 

This particular study answers some intriguing questions about how the incentives 
played out in the daily lives of teenage children and their parents. The study surveyed a subset 
of families from the Family Rewards’ core sample who had a teenage child and collected 
infonnation both from parents and teenagers. It examines some key mechanisms that may 
underlie the program’s effects on the targeted educational outcomes, such as whether there were 
changes in how teenagers spent their after-school and evening time. It also looks at how the 
program affected outcomes that were not targeted by the program, such as teenagers’ aggression 
and use of drugs and alcohol, and whether the program produced unintended consequences, 
such as increasing parent-child conflict or teenagers’ anxiety or depression. 

We found some particularly interesting results. The program increased the proportion of 
teenagers who spent their time in academically oriented ways, especially among the students 
who were better prepared for high school when they entered the study — the same group who 
had better overall academic outcomes as a result of Family Rewards. Unfortunately, there was 
no similar shift toward spending time on academic activities among less-proficient students. 
And overall, students did not report being more engaged in school or having a greater sense of 
efficacy about their academic competence. 
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In addition, parents were more likely to save for their child’s future education and spend 
more on educational resources. The teens in the program group reported lower levels of aggres- 
sion and substance use than their peers in the control group, which could bode well for their 
long-term outcomes. And concerns that the program would cause greater parent-child conflict 
and anxiety among teenagers or reduce their intrinsic motivation to do well in school were not 
borne out. 

This study makes a valuable contribution to the evaluation by shedding light on the 
ways in which the program functioned to effect change and suggesting some promising design 
directions for the next generation of CCT tests. 


Gordon L. Berlin 
President 
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Executive Summary 


In 2007, the Center for Economic Opportunity (CEO) in the Mayor’s Office of the City of New 
York launched the Opportunity NYC-Family Rewards Demonstration (referred to as Family 
Rewards in this report), a conditional cash transfer (CCT) program. Family Rewards is based on 
the rationale of CCT programs such as Mexico’s well-known Oportunidades model, but has 
been adapted in specific ways to meet the needs of families in New York City. CCT programs 
offer cash assistance to reduce immediate poverty among low-income families, while condition- 
ing that assistance on parents’ investment in their children’s human capital (for example, 
adherence to well-baby visits, immunizations, or school attendance) with the goal of breaking 
the intergenerational cycle of poverty. This report presents the results of an innovative study 
designed to provide a detailed understanding of the ways in which families and their teenage 
children were affected by Family Rewards, one of the first comprehensive CCT programs to be 
tested in a developed country. 

Family Rewards offered cash rewards in three key areas thought to be critical to fami- 
lies’ short- and long-tenn success: education, health care, and parents’ employment. The 
program was offered to low-income families with children in the fourth, seventh, or ninth 
grades in six of the poorest neighborhoods of New York City. Two national, New York-based 
nonprofit organizations — MDRC, a social policy and education research finn, and Seedco, a 
workforce and economic development organization — worked in close partnership with CEO 
to design the demonstration. The three-year program began in September 2007 and ended in 
August 2010. 

MDRC is currently conducting an extensive evaluation of the Family Rewards pro- 
gram. To assess the effects, or “impacts” of the program, parents and their children were offered 
the opportunity to take advantage of the CCTs for three years as part of the program group, or to 
be part of a control group that was not offered the CCTs. The focus of the “core study” is on the 
program’s implementation as well as its impacts across a wide range of outcome measures 
likely to be affected by the program. As described in the first report on interim findings from the 
core study, 1 most families received reward payments for meeting the conditions the program 
established, and these payments amounted to a substantial sum. As a result, Family Rewards 
was found to reduce current poverty and material hardship and produced a range of effects on a 
variety of outcomes across all three program domains (children’s education, family health care, 
and parents’ work and training). While the program did not improve school outcomes overall 


Tames Riccio, Nadine Dechausay, David Greenberg, Cynthia Miller, Zawadi Rucks, and Nandita Verma, 
Toward Reduced Poverty’ Across Generations: Early Findings from New York City ’s Conditional Cash 
Transfer Program (New York: MDRC, 2010). 


ES-1 



for elementary or middle school students, it did increase academic outcomes for children who 
were better prepared for high school when they entered the study in the ninth grade. The 
children who entered the program in the ninth grade and their parents are the focus of this 
particular report. 

The findings reported in this special “embedded” study add infomiation about families’ 
interactions with the CCT program in key areas by interviewing teenagers and their parents 
approximately two years after they enrolled in the program. More specifically, this report 
presents the impacts of the Family Rewards program on (1) the way teenagers spent their time; 
(2) the way teenagers and parents spent their money; (3) interactions between parents and 
teenagers; (4) teenagers’ academic engagement, motivation, and sense of competence in school; 
and (5) teenagers’ mental health and problem behaviors, such as aggression or substance use. 
As such, this report presents results on key “mechanisms of action” that may underlie any 
effects (or, indeed, lack of effects) of the program on targeted outcomes (such as test scores) as 
well as key outcomes that were not targeted but could be affected nonetheless by the program. 

This study found that Family Rewards affected some of the channels by which the pro- 
gram is thought to operate. Most notably, Family Rewards changed how teenagers spent their 
time, increasing the proportion of teenagers who engaged primarily in academic activities. 
These impacts on the way teenagers allocated their time were concentrated among those who 
were more academically prepared for school at the start of the study. This same group of more 
proficient teenagers had experienced gains in academic outcomes as a result of the program. 2 
Family Rewards also changed how teenagers and their parents spent their money. For example, 
parents enrolled in Family Rewards reported saving more for their children’s future education. 
However, other pathways by which the program was thought to operate — for families (parent- 
teenager interactions) and for teenagers themselves (school engagement and academic efficacy) 
— were not affected by the program. 

With regard to outcomes not targeted directly by the program, the news was quite posi- 
tive. First, Family Rewards did not increase parent-teenager conflict or cause anxiety or depres- 
sion among teenagers — among the potential unintended consequences of the program that 
were of concern. Second, Family Rewards was found to reduce teenagers’ aggressive behavior 
and substance use. 


The Family Rewards Program 

Family Rewards provided cash payments to families who successfully carried out and docu- 
mented a number of health, education, and work activities. The program offered a set of 22 

2 Riccio et al. (2010). 
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different incentives during its first two years and a smaller number in the third year, ranging in 
value from $20 to $600.' These incentives are described in Table ES.l. The incentives encour- 
aged families to meet conditions in three domains: health, education, and work. 

• Health-focused conditions included maintaining health insurance coverage 
for parents and their children as well obtaining age-appropriate preventive 
medical and dental checkups for each family member. 

• Education-focused conditions included meeting goals for children’s attend- 
ance in school, achievement levels on standardized tests, and other school 
progress markers, as well as parents’ engagement with their children’s educa- 
tion. 

• Workforce-focused conditions, aimed at parents, included sustaining full- 
time work and participating in approved education or job training activities. 

Every two months, the program verified that families had earned rewards by using a 
combination of automated data from city agencies and special forms, or “coupons,” which the 
participants completed and submitted directly to the program. Some rewards were offered to the 
oldest children themselves (rather than to their parents), meaning that these children were in 
principle more directly exposed to incentives than younger children. 


Study Design and Sample 

The Family Rewards core evaluation involves approximately 4,800 families and 11,000 
children who applied to the program. The program was targeted to families who lived in 
selected high-poverty community districts and who had incomes at or below 130 percent of the 
federal poverty level. Eligible families had to have at least one child in the fourth, seventh, or 
ninth grade. These grades were selected because they are at or near the start of critical transition 
points in children’s educational careers. Once a family volunteered for the study, all children in 
the family who were school age or younger were eligible for the program. Through a lottery- 
like process, half of the applicant families were selected for Family Rewards and offered the 
incentives discussed above, and half were assigned to a control group that was not offered the 
incentives. 

This report focuses on a subset of the children in the oldest age cohort (ninth grade at 
baseline) from the Family Rewards core study sample who were interviewed as part of the core 


Tn Year 3 of the program, some rewards were eliminated in response to preliminary findings, verification 
difficulties, and the need to ensure that funding could continue for a third year. See Chapter 1 in Riccio et al. 
(2010), pages 17-27, for a more extensive discussion of the rewards structure and changes that were made to it. 
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Table ES.l 

Schedule of Rewards 


Activity 

Reward Amount 

Education incentives 


Elementary and middle school students 


Attends 95% of scheduled school days (discontinued after Year 2) 

$25 per month 

Scores at proficiency level (or improves) on annual 
math and English Language Arts (ELA) tests 
Elementary school students 
Middle school students 

$300 per math test; $300 per ELA test 
$350 per math test; $350 per ELA test 

Parent reviews low-stakes interim tests ( discontinued after Year 1) 

Parent discusses annual math and ELA test results with 
teachers (discontinued after Year 2) 

$25 for parents to download, print, and review 
results (up to 5 times per year) 

$25 (up to 2 tests per year) 

High school students 


Attends 95% of scheduled school days 
Accumulates 1 1 course credits per year 
Passes Regents exams 
Takes PS AT test 
Graduates from high school 

$50 per month 
$600 

$600 per exam passed (up to 5 exams) 
$50 for taking the test (up to 2 times) 
$400 bonus 

All grades 


Parent attends parent-teacher conferences 

Child obtains library card ( discontinued after Year 2) 

$25 per conference (up to 2 times per year) 
$50 once during program 

Health incentives 


Maintaining public or private health insurance (discontinued after Year 2) 
For each parent covered 
If all children are covered 

Per month: $20 (public); $50 (private) 
Per month: $20 (public); $50 (private) 

Annual medical checkup 

$200 per family member (once per year) 

Doctor-recommended follow-up visit (discontinued after Year 2) 

$100 per family member (once per year) 

Early-intervention evaluation for child under 30 months old, 
if advised by pediatrician 

$200 per child (once per year) 

Preventive dental care (cleaning/checkup) 

$100 per family member (once per year for 
children 1-5 years old; twice per year for 
family members 6 years of age or older) 

Workforce incentives 


Sustained full-time employment 

$150 per month 

Education and training while employed at least 10 hours per week 
(employment requirement discontinued after Year 2) 

Amount varies by length of course, up to a 
maximum of $3,000 over three years 
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evaluation follow-up at the 18-month time point. A sample of 716 teenagers and their parents 
were selected from this group of families, and information was collected as part of this embed- 
ded study on 5 1 1 of the teenagers — those whose parents agreed to allow them to participate 
and who completed the survey. 

A telephone survey was administered to the teenagers and their parents during the 
spring of their third year following random assignment (approximately 30 months after they 
entered the core study). The surveys collected information from teenagers on their spending and 
saving patterns, time use, level of conflict in discussions of various topics with their parents, 
various educational processes, mental health, and risky behaviors. The parent surveys collected 
information from parents on their spending and saving patterns, levels of monitoring their 
teenager, and level of conflict in discussions of various topics with their teenager. 

Analyses presented in the current report focus on the differences between parents and 
teenagers assigned to the program group and those assigned to the control group in each of the 
areas of interest discussed above, controlling for a small set of pre-random-assignment charac- 
teristics to increase precision. 4 


Impacts of the Family Rewards Program 

The evidence emerging from this embedded study is encouraging and shows that some behav- 
iors of parents and children can be changed for the good by the CCT incentives offered by the 
Family Rewards program. And in cases where concerns were raised that the program might 
harm families (for example, increase parent-teenager conflict and anxiety on the part of the 
teenagers or decrease teenagers’ intrinsic motivation), there was no evidence that Family 
Rewards had “unintended consequences.” At the same time, a number of plausible processes 
that could be activated as a result of the program were not affected, highlighting areas that 
might be directly targeted in future interventions that include CCTs. 

Five main conclusions emerge from this study of the Family Rewards program: 

• Family Rewards increased the proportion of teenagers who spent their 
time predominantly in academic activities. 


4 A range of baseline covariates were included in all analyses. Covariates included average class size for 
grade level, ELA pretest score, math pretest score, school district, child gender, child race, special education 
status, the number of children in the household, whether English was spoken in the home, whether the family 
was a single-parent or two-parent family, parents’ education, parents’ employment status, an indicator for if 
there was no mother in the home, and an indicator for whether random assignment occurred after September 
2007. 
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At the outset of Family Rewards, it was hoped that teenagers would spend more time in 
school-related activities (such as completing homework assigmnents and studying) and in 
achievement-oriented after-school activities (such as tutoring), because the outcomes of such 
behaviors (that is, test scores) were directly incentivized by the program. 

Based on an analysis of teenagers’ reported afternoon and evening activities, teenagers 
were divided into four groups: (1) those who spent their time predominantly in academically 
oriented activities (such as homework, after-school activities); (2) those who spent their time 
predominantly in socially oriented activities (such as hanging out with friends or family, talking 
or texting on the phone; (3) those who spent a large proportion of time watching TV and on the 
computer; and (4) those who spent the most time in maintenance activities (eating, napping, 
grooming, commuting) as well as had the highest level of work (chores or work for pay) of all 
the groups. 

As shown in Figure ES.l, Family Rewards did indeed lead to a shift in the way teenag- 
ers allocated their time. More specifically, Family Rewards reduced the proportion of teenagers 
who spent their time predominantly in social activities (from 2 1 percent in the control group to 
15 percent in the program group) and increased the proportion of teenagers who spent their time 
predominantly in academic activities (from 16 percent in the control group to 24 percent in the 
program group). 5 Separate analyses not shown here indicate that this finding was especially true 
for the teenagers who were better prepared for school based on their standardized test scores 
before random assigmnent; notably, it is these same teenagers who were found in the core study 
to demonstrate gains in test scores as a result of the Family Rewards program. 

• Family Rewards increased parents’ spending on their teenagers and 
saving for their teenagers’ future education, indicating positive effects 
on parents’ investment in children’s human capital. 

By providing cash transfers to families, Family Rewards should have resulted in greater 
spending and, perhaps, saving within families. However, whether or not those increased 
resources would be spent in ways that would benefit children’s long-term human capital was an 
open question. Thus, the embedded study was intended to examine the impacts of the Family 
Rewards program on parents’ and teenagers’ spending and savings. Results are shown in Table 
ES.2. Parents in the program group had higher levels of monthly spending on academic activi- 
ties and supplies (of $31) and on leisure and entertainment (of $27). Notably, Family Rewards 
also had a positive impact on parents’ indicating that they were currently saving for their 


5 As discussed in the report, while Family Rewards did increase the proportion of teens in an academically 
oriented group, there were no impacts on the average amount of time spent in academic activities across all 
teenagers. 
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The Opportunity NYC Demonstration: Family Rewards 
Figure ES.l 

Impacts on Teenagers' Membership in Time Use Groups 
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SOURCE: MDRC calculations from the Family Rewards 30-Month Embedded Study Survey. 

NOTE: Estimates were regression-adjusted using ordinary least squares, controlling for pre-random 
assignment characteristics of family and sample members. 


teenager’s future education, a sizeable 12.6 percentage point increase. Teens in the Family 
Rewards group spent somewhat less on necessities and leisure activities than control group 
teens, perhaps because parents were spending more. Notably, these findings should be consid- 
ered in relation to the timing of the embedded study survey, which was fielded in the late winter 
and spring, before the large rewards that teenagers could earn were available that year. 

• Family Rewards did not affect interactions between parents and teenag- 
ers, for better or for worse. 

Family Rewards was expected to change parent- teenager interactions in a number of 
ways. On the one hand, rewards might shape families’ norms and expectations about children’s 
participation in school and after-school activities, and, as a result, parents might spend more 
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The Opportunity NYC Demonstration: Family Rewards 

Table ES.2 


Impacts on Spending and Saving and Parent-Teenager Interactions 


Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P -Value 

Spending and saving 

Total parent spending in 1 month 11 ($) 

Total spending 

695 

624 

71 * 

0.056 

Productive spending 

156 

125 

32 *** 

0.007 

Maintenance spending 

434 

418 

16 

0.552 

Leisure spending 

109 

82 

27 *** 

0.008 

Total teenager spending in 1 month b ($) 

Total spending 

230 

301 

-72 ** 

0.019 

Productive spending 

34 

38 

-4 

0.600 

Maintenance spending 

162 

213 

-51 ** 

0.020 

Leisure spending 

35 

51 

-16 * 

0.070 

Save for child's future education (%) 

54.1 

41.5 

12.6 *** 

0.005 

Parent-teenager interactions 

Intensity level of family conflict (5 = angry, 1 = calm) 

Parent report 

2.0 

2.1 

0.0 

0.832 

Teenager report 

2.0 

2.0 

-0.1 

0.296 

Monitoring/disclosure 

Parental monitoring (5 = high monitoring) 

4.3 

4.3 

0.0 

0.928 

Teenager disclosure to parent (5 = high disclosure) 

3.9 

3.8 

0.1 

0.495 

Sample size (total = 511) 

262 

249 




SOURCE: MDRC calculations from the Family Rewards 30-Month Embedded Study Survey. 

NOTES: Statistical significance levels indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre-random assignment 
characteristics of family or sample members. 

Rounding may cause slight discrepancies in calculating sums and differences. 

Sample sizes may vary because of missing values. 

A two-tailed t-test was applied to the differences between outcomes for the program and control groups. 

The p-value indicates the likelihood that the difference between the program and control group arose by chance. 
“Spending on different items was reported for varying time periods ranging from 1 to 6 months. All numbers 
reported in this table are scaled to a 1 -month interval. Productive spending = educational and other structured 
activities and materials; maintenance spending = basic necessities; leisure spending = "fun" money. 
b Spending on different items was specifically asked about "in the past 4 weeks." 

time being involved in and monitoring their children’s activities in and out of school. At the 
same time, designers expressed concern that Family Rewards, by introducing incentives to 
families, might increase parent-teenager conflict during an already tumultuous period in 
families’ lives. As shown in the bottom panel of Table ES.2, however, neither the expected 
benefits nor the concerns about the program were borne out. That is, parents were not found to 


ES-8 






engage in greater monitoring of their teenager’s behavior, as might be expected if Family 
Rewards was making large changes in how families interacted with their children. Neither was 
there any indication that Family Rewards increased parent-child conflict. 

• Family Reward did not affect teenagers’ approaches to schooling, 
measures thought to mediate any benefits to teenagers’ academic out- 
comes. 

A central premise of the Family Rewards program was that by offering educational in- 
centives for attendance and achievement outcomes, children’s interest in school and their sense 
of competence in academics would increase. At the same time, designers worried that by 
providing external rewards for academic achievement, the program could undermine intrinsic 
motivation. As shown in Table ES.3, students did not report greater engagement in school or a 
greater sense of efficacy about schoolwork. On the positive side, the provision of rewards did 
not appear to undermine intrinsic motivation, a central concern about this program at the outset. 

• Family Rewards reduced teenagers’ problem behaviors, key outcomes 
not directly targeted by the intervention. No increases were found in 
teenagers’ depression or anxiety. 

In addition to effects on the key targets of the program, this study was intended to ad- 
dress the possible effects of Family Rewards on teenagers’ mental health and problem behav- 
iors. The effects on these nontargeted outcomes are critical to understanding the broader effects 
of the program. As shown in the bottom panel of Table ES.3, teenagers in the program group 
reported lower levels of aggression (a 10 percentage point reduction) and lower levels of 
substance use (an overall reduction of 15 percentage points). They also reported reductions in 
alcohol and marijuana use of 14 and 7 percentage points, respectively (shown in Chapter 3 of 
the full report in Table 3.2), compared with their peers in the control group. Unlike the findings 
on academic outcomes and time use, these findings on substance use do not differ for children 
who were more or less academically proficient when they entered the study. These findings 
bode well for teenagers’ longer-term outcomes. And as shown in the middle panel of Table 
ES.3, there are no statistically significant impacts on teenagers’ depression and anxiety. Given 
concerns that parents would put undue pressure on their children and cause increased anxiety, 
this “null” finding should be seen as good news for the program. 
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The Opportunity NYC Demonstration: Family Rewards 
Table ES.3 


Impacts on Teenagers' Approaches to Schooling, 
Mental Health, and Problem Behaviors 


Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P -Value 

ADDroaches to schooling 

Think will finish college 

85.8 

84.3 

1.6 

0.608 

School behavioral engagement 

4.3 

4.3 

0.0 

0.557 

Academic efficacy 

2.9 

2.9 

0.1 

0.252 

Motivation to learn (- 7 = extrinsic, 7 = intrinsic) 

0.7 

0.5 

0.2 

0.421 

Mental health outcomes 

Depression (0 to 13 scale) 

1.6 

1.5 

0.1 

0.557 

Anxiety (0 to 13 scale) 

3.6 

4.0 

-0.4 

0.148 

Problem behaviors 

Delinquent behaviors (%) 

46.1 

52.7 

-6.6 

0.141 

Aggressive behaviors (%) 

14.1 

24.1 

_9 9 *** 

0.004 

Any substance use in past month (%) 

17.5 

32.6 

-15 2 *** 

0.000 

Number of friends using substances in past month 

( 1 = none, 2 = a few, 3 = some, 4 = most) 

1.81 

1.99 

-0 19 *** 

0.004 

Sample size (total = 511) 


262 

249 



SOURCE: MDRC calculations from the Family Rewards 30-Month Embedded Study Survey. 

NOTES: Statistical significance levels indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 
percent. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre-random 
assignment characteristics of family or sample members. 

Rounding may cause slight discrepancies in calculating sums and differences. 

Sample sizes may vary because of missing values. 

A two-tailed t-test was applied to the differences between outcomes for the program and control 
groups. 

The p-value indicates the likelihood that the difference between the program and control group arose 
by chance. 


Conclusion 

Family Rewards can be viewed as “CCT 1.0” in the United States. CCTs in Latin America and 
elsewhere in the global South have often been based on pilot studies meant to identify obstacles 
to take-up of health, nutrition and education services and to estimate the size and timing of 
incentives that would most efficiently and effectively improve use of these services. Because 
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Family Rewards was designed and implemented under a tight timeline, no such pilot studies 
were conducted in New York. Hence, the results of the evaluations of Family Rewards, both the 
core evaluation and this embedded study, can be used to design “CCT 2.0.” For these reasons, 
this study is more of a theory-building and hypothesis-generating study than a theory- 
confirming, hypothesis-testing study. 

As discussed in detail in Riccio et al. (2010), the Family Rewards program had modest 
impacts on high school students’ academic achievement, but only for a portion of the sample 
(those children who scored at proficiency levels on baseline test scores). The results of this 
embedded study shed light on some hypotheses to explain why this occured. There were small 
changes in the amount of time students spent on academic activities in place of social activities, 
but only for the academically proficient teenagers. There were also changes in the way families 
allocated their increased financial resources. However, there are no statistically significant 
impacts on key hypothesized mediators among families, such as parental monitoring. Moreover, 
key student-level academic processes, such as perceived academic efficacy as well as academic 
expectations, were unchanged by the program. 

These results lead to some suggestions for future work. First, more attention needs to be 
paid to redesigning incentives that would be effective for nonproficient teenagers. Second, 
future CCT initiatives in the United States also might consider redesigning incentives to directly 
target pathways of influence in order to produce larger and more sustained impacts on out- 
comes. For example, rather than incentivizing academic achievement as measured by standard- 
ized tests, rewards could offer students incentives for grade point averages or homework. Or 
they could encourage students to attend academic tutoring programs or to join academic clubs, 
which have been shown to improve academic achievement and might change the way students 
approach schooling more effectively. 6 

It will be critical to examine the effects of the Family Rewards program in the long 
term. The effects on proficient students’ activity participation and on parents’ savings for their 
children’s future education, combined with the reductions in teenagers’ aggression and sub- 
stance use, point to potential long-term benefits of the program for teenagers as they move into 
adulthood. Future reports on this study will present findings on the program’s impacts at later 
points in time — one will examine impacts three years after random assigmnent, around the 
time that the program itself was ending, and the final report will examine results over a five- 
year follow-up period, including two years after the program ended. The postprogram follow-up 
period will allow the evaluation to determine the long-term implications of the findings from 
this embedded study. 


6 J. D. Coie and G. Krehbiel, “Effects of Academic Tutoring on the Social Status of Low- Achieving, So- 
cially Rejected Children,” ( Child Development 55:1465-1478 [1984]). 
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Chapter 1 

Introduction 


Background 

In 2007, the Center for Economic Opportunity (CEO) in the Mayor’s Office of the City of New 
York mounted Opportunity NYC-Family Rewards (referred to as Family Rewards in this 
report), a conditional cash transfer (CCT) program. Family Rewards was based on the rationale 
of CCT programs such as Mexico’s well-known Oportunidades model but was adapted in 
specific ways to meet the needs of families in New York City. CCT programs offer cash 
assistance to reduce immediate poverty among low-income families, while conditioning that 
assistance on parents’ investment in their children’s human capital (for example, adherence to 
well-baby visits, immunizations, or school attendance) with the goal of breaking the intergener- 
ational cycle of poverty. There is evidence that such programs have had some success in 
developing countries. 1 However, Family Rewards was the first comprehensive CCT initiative 
implemented and tested in a developed country. 

Family Rewards offered cash rewards in three key areas thought to be critical to fami- 
lies’ short- and long-tenn success: children’s education, family preventive health care, and 
parents’ work and training. The program targeted low-income families with children in the 
fourth, seventh, and ninth grades (but all eligible children in the family were enrolled, regardless 
of their grade level) in six of the poorest neighborhoods of New York City. MDRC, a nonprofit 
social and education policy research finn, designed and is conducting a rigorous random 
assignment evaluation of the program, in which parents and their children were offered the 
opportunity to take advantage of CCTs for three years or to be part of a control group that was 
not offered CCTs. The three-year program began in September 2007 and ended in August 2010. 

This report presents the results of a “child and family embedded study” — that is, a 
study of teenagers and their parents embedded in the core evaluation being conducted by 
MDRC. The core evaluation represents an ongoing and comprehensive study of the effects of 
Family Rewards on families. The focus of the core study is on the program’s implementation as 
well as its effects, or “impacts” across a wide range of outcome measures likely to be affected 
by the program. Data were gathered from three key sources of information: (1) administrative 
sources of data (school records, welfare benefits, unemployment insurance records); (2) a 
survey of parents; and (3) qualitative interviews conducted with parents and their teenage 


'Fiszbein and Schady (2009). 
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children participating in the Family Rewards program. In formation from the core study has 
been released in a series of early reports on the project. 2 

As described in the first report on interim findings from the core evaluation, 3 most fami- 
lies received reward payments and a substantial amount of money for meeting the conditions 
the program established. Rigorous analyses comparing the outcomes of families randomly 
assigned to the Family Rewards program with those of families assigned to the control group 
show that Family Rewards reduced current poverty and material hardship. It also produced a 
range of effects on a variety of outcomes across all three program domains (children’s educa- 
tion, family preventive health care, and parents’ work and training). Impacts on academic 
outcomes for children were concentrated among the oldest children — those in ninth grade who 
are the focus of this report — and among these students, those who were most prepared aca- 
demically at the start of the study. Analysis of the longer-term effects of the program is current- 
ly under way. 


Current Study 

This embedded study adds information in a number of key areas of families’ lives by interview- 
ing the oldest of the focal children — those who were in ninth grade when they entered the 
Family Rewards core study — as well as their parents, two years after they entered the study. 
More specifically, this report presents the impacts of the Family Rewards program on (1) the 
way teenagers spent their time; (2) the way teenagers and parents spent their money; (3) 
interactions between parents and teenagers; (4) teenagers’ academic engagement, motivation, 
and sense of competence in school; and (5) teenagers’ mental health and problem behaviors 
(such as aggression and substance use). As such, this report presents results on key “mecha- 
nisms of action” that may underlie any effects (or indeed, lack of effects) of the program on 
targeted outcomes (such as test scores) as well as key outcomes that were not targeted but that 
may be affected nonetheless by this program (such as problem behaviors). Indeed, this study 
was conceived to test those processes within families and children that might represent key 
pathways of influence of the program. The hope was that the infomiation from this study might 
shed light on processes underlying any positive effects of the program as well as potentially 
countervailing influences that might explain neutral effects on targeted outcomes. 

As mentioned above, the core evaluation found impacts on academic outcomes for the 
ninth-grade children who were most prepared academically at the start of the study. 4 Findings 
from this embedded study show that Family Rewards affected some but not all of the channels 

2 Riccio et al. (2010); Greenberg, Dechausay, and Fraker (2011). 

3 Riccio et al. (2010). 

4 Riccio et al. (2010). 
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by which the program is thought to operate. Most notably, Family Rewards changed how 
teenagers spent their time, increasing the proportion of teenagers who engaged primarily in 
academic activities. These impacts on time use were especially pronounced for those who were 
more academically prepared for school at the start of the study. Family Rewards also changed 
how teenagers and their parents spent their money. For example, parents enrolled in Family 
Rewards reported saving more for their children’s future education. However, other mecha- 
nisms by which the program was thought to operate — both for families (parent-teenager 
interactions) and for teenagers themselves (school engagement and academic efficacy) — were 
not affected by the program. 

With regard to outcomes not targeted directly by the program, the news was quite posi- 
tive. First, Family Rewards did not adversely affect outcomes such as mental health that were of 
concern as potential unintended consequences of the program. Second, Family Rewards was 
found to reduce teenagers’ aggressive behavior and substance use, which may bode well for 
their longer-term outcomes. 

In the next section of this chapter, the Family Rewards program is placed in the context 
of research on other CCT programs, more specifically, and financial incentive programs, more 
broadly. Following this discussion, infonnation is presented on the operation of the Family 
Rewards program and the interim impacts from the core evaluation, as context for understand- 
ing the findings presented here. Turning to the current study, the theory of change underlying 
this embedded study and an overview of the sampling and analytic approach for the study are 
presented. Subsequent chapters present findings from this embedded study. 


Placing the Family Rewards Program in Context 

Comprehensive CCTs have been developed and implemented over the last 15 years, first in 
Latin America and increasingly in Africa and Asia. In their review of the first wave of well- 
evaluated CCTs, Fiszbein and Schady (2009) concluded that CCTs can reduce poverty, increase 
the use of health, nutrition, and education services, and improve health outcomes, especially in 
early childhood. But there is less evidence that CCTs have an impact on children’s learning and 
academic achievement, despite their positive impacts on school enrollment and attendance. 

Comprehensive CCTs are one variant of the broader use of financial incentives to 
change behavior. Other types of financial incentives designed to change the health, education 
and/or work behaviors, and outcomes of low-income people also have been evaluated over the 
last two decades. Considerable variation exists among these nonholistic incentive systems. For 
example, most target change in only one domain (such as health, education, or work), some use 
noncash incentives (for example, a time-deferred scholarship), and the value of the incentives 
can vary widely. And just as the target and design of the incentives vary, so too do the results. 
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Conceptually, one would not expect domain-specific incentives of less value to have the same 
impact on behavior and outcomes as incentive systems that target several domains and are of 
more value. Thus, comprehensive CCTs are expected to have more synergistic effects across 
several domains of behaviors and outcomes. 

This broader literature on the use of financial incentives to motivate change in the 
health, education, and work behaviors and outcomes of low-income populations can serve as a 
useful context within which to discuss the design and impact of Opportunity NYC-Family 
Rewards. 

Family Rewards can be viewed as “CCT 1.0” in the United States. CCTs in Latin 
America and elsewhere in the global South have often been based on pilot studies meant to 
identify obstacles to take-up of health, nutrition, and education services and to estimate the size 
and timing of incentives that would most efficiently and effectively improve use of these 
services. Because Family Rewards was designed and implemented under a tight timeline, no 
such pilot studies were conducted in New York. Hence, the results of the evaluations of Family 
Rewards, both the core evaluation and this embedded study, may be used to design “CCT 2.0.” 
Indeed, that is just what is happening. New York City’s CEO and MDRC are designing and 
launching a second-stage version of Family Rewards with funds from the White House’s Social 
Innovation Fund, as part of MDRC’s and CEO’s Social Innovation Fund Project. For these 
reasons, this embedded study should be viewed more as a theory-building and hypothesis- 
generating study than a theory-confirming, hypothesis-testing study. This issue will be dis- 
cussed further in the conclusion. 


The Operation of the Family Rewards Program 

The Family Rewards Demonstration provided cash payments to families who successfully 
carried out and documented a number of health, education, and work activities. The demonstra- 
tion was one of 40 initiatives sponsored by New York City’s CEO, a unit within the Office of 
Mayor Michael R. Bloomberg that is responsible for testing innovative strategies to reduce the 
number of New Yorkers who are living in poverty. Its broad origins were in the CCT move- 
ment, which has demonstrated success in lower- and middle-income countries in improving 
human capital and reducing hardship. Two national, New York-based nonprofit organizations 
— MDRC, a nonpartisan social policy and education research finn, and Seedco, a workforce 
and economic development organization — worked in close partnership with CEO to design the 
demonstration. Seedco, together with a small network of local community-based organizations, 
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operated Family Rewards, while MDRC managed the overall demonstration and is conducting 
the evaluation. A consortium of private funders supported the project. 5 

Family Rewards offered a set of 22 different incentives during its first two years and a 
smaller number in the third year, ranging in value from $20 to $600. 6 These incentives are 
described in Table 1.1. These incentives encouraged families to meet conditions in three 
domains: education, health, and work. For example, education rewards for younger students 
involved $25 per month for attending 95 percent of scheduled school days, $25 per conference 
for one or both parents to attend parent-teacher conferences, and $300 to $350 per exam for 
passing standardized tests in English Language Arts (ELA) or in math, or for demonstrating 
improvement in them. For high school students, the rewards were higher. They involved $50 
per month for attendance, $600 for accumulating sufficient credits to move into the next grade, 
$600 for each New York State Regents exam passed (a standardized test tied to graduation), 
$50 for taking the Preliminary SAT (PSAT), and $400 for graduating from high school. 
Rewards for health activities included payments for going to the dentist, for annual medical 
checkups, and for follow-up visits recommended by physicians. Regarding work, under the 
theory that full-time work was necessary to help families escape from poverty, families were 
given $150 per month for working at least 30 hours a week for six of eight weeks and a reward 
for completing training, depending on the duration of the class. 

Every two months, Seedco verified that families had earned rewards by using a combi- 
nation of automated data from city agencies and special forms, or “coupons,” which the partici- 
pants filled out and submitted directly to the program. Operators then initiated a process of 
transferring payments electronically into participants’ bank accounts or into accounts that 
operate more like gift cards, which were set up depending on the participants’ preference when 
they entered the program. Reward payments were then made, and families could access their 
money at any time. A significant feature of implementation was that most rewards for the older 
children went directly into their own accounts, meaning that these children were, in principle, 
more directly exposed to incentives than younger children (whose parents received rewards for 
their achievement). This cycle of completing activities, submitting any relevant coupons, 
verification by program staff, and payment to the participant was repeated every two months. 


5 The funders include Bloomberg Philanthropies, The Rockefeller Foundation, The Starr Foundation, the 
Open Society Institute, the Robin Flood Foundation, the Tiger Foundation, The Annie E. Casey Foundation, 
American International Group, the John D. and Catherine T. MacArthur Foundation, and New York Commu- 
nity Trust. 

6 ln Y ear 3 of the program, some rewards were eliminated in response to preliminary findings, verification 
difficulties, and the need to ensure that funding could continue for a third year. See Chapter 1 in Riccio et al. 
(2010), pages 17-27, for a more extensive discussion of the rewards structure and changes that were made to it. 
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The Opportunity NYC Demonstration: Family Rewards 

Table 1.1 

Schedule of Rewards 


Activity 

Reward Amount 

Education incentives 


Elementary and middle school students 


Attends 95% of scheduled school days (discontinued after Year 2) 

$25 per month 

Scores at proficiency level (or improves) on annual 
math and English language arts (ELA) tests 
Elementary school students 
Middle school students 

$300 per math test; $300 per ELA test 
$350 per math test; $350 per ELA test 

Parent reviews low-stakes interim tests (discontinued after Year 1) 

Parent discusses annual math and ELA test results with 
teachers (discontinued after Year 2) 

$25 for parents to download, print, and review 
results (up to 5 times per year) 

$25 (up to 2 tests per year) 

High school students 


Attends 95% of scheduled school days 
Accumulates 1 1 course credits per year 
Passes Regents exams 
Takes PSAT test 
Graduates from high school 

$50 per month 
$600 

$600 per exam passed (up to 5 exams) 
$50 for taking the test (up to 2 times) 
$400 bonus 

All grades 


Parent attends parent-teacher conferences 

Child obtains library card ( discontinued after Year 2) 

$25 per conference (up to 2 times per year) 
$50 once during program 

Health incentives 


Maintaining public or private health insurance (discontinued after Year 2) 
For each parent covered 
If all children are covered 

Per month: $20 (public); $50 (private) 
Per month: $20 (public); $50 (private) 

Annual medical checkup 

$200 per family member (once per year) 

Doctor-recommended follow-up visit (discontinued after Year 2) 

$ 1 00 per family member (once per year) 

Early-intervention evaluation for child under 30 months old, if advised by 
pediatrician 

$200 per child (once per year) 

Preventive dental care (cleaning/checkup) 

$100 per family member (once per year for 
children 1-5 years old; twice per year for 
family members 6 years of age or older) 

Workforce incentives 


Sustained full-time employment 

$150 per month 

Education and training while employed at least 10 hours per week 
(employment requirement discontinued after Year 2) 

Amount varies by length of course, up to a 
maximum of $3,000 over three years 


6 



One notable feature of Family Rewards was its “incentives only” design. As mentioned, 
the program was designed to test the feasibility of a model that did not rely on services, so as to 
devote more resources directly to low-income families and to make it potentially possible to 
replicate the program on a larger scale. After participants were recruited by community-based 
organizations (known as Neighborhood Partner Organizations, or NPOs), they were asked to 
return for orientations during which they were introduced to the incentives and assisted in 
creating new bank accounts or linking existing accounts to program payments. However, after 
these orientations, the participants had no mandated contact with Family Rewards. They 
received mailings related to their bimonthly program earnings and were targeted by marketing 
efforts that highlighted eligible reward activities in materials produced by an advertising 
agency. They had access to a Web site and received automated and staffed reminder calls. 
Participants were also invited to attend topical workshops and could call either the NPOs or a 
centralized “helpline” during extended business hours for help with problems related to pay- 
ments or for advice about services in the community. However, workshops were generally 
underattended, and participants used the NPOs infrequently as a referral resource. As a result, 
participants’ contact with the program could be quite limited — it was, in fact, possible for 
families to receive rewards with little contact. 


Core Evaluation: Design and Early Findings 

The Family Rewards core evaluation involves approximately 4,800 families and 11,000 
children who applied to the program. The program was targeted to families who lived in 
selected high-poverty community districts and who had incomes at or below 130 percent of the 
federal poverty level. Eligible families had to have at least one child in the fourth, seventh, or 
ninth grade. These grades were selected because they are at or near the start of critical transition 
points in children’s educational careers. Once a family volunteered for the study, all children in 
the family who were school age or younger were eligible for the program. Through a lottery- 
like process, half of the applicant families were selected for Family Rewards and offered the 
incentives discussed above, and half were assigned to a control group that was not offered the 
incentives. 

Early results reported in March 2010 showed that despite an extraordinarily rapid start- 
up and early challenges, Family Rewards was operating largely as intended by its second year. 
Although many families struggled with the complexity of the program, most received a large 
amount of money for meeting the conditions it established — on average, families received 
approximately $6,000 during the first two years of the program. During the period covered by 
the report, Family Rewards reduced poverty and hardship and produced a range of effects on a 
variety of outcomes related to health care, parents’ work and training, and children’s education. 
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Interim impacts of the Family Rewards program from the core study (the estimated dif- 
ference between those assigned to the program group and those assigned to the control group, 
controlling for a small set of pre-random-assignment covariates) are summarized in Table 1.2. 
(See Box 1.1 for infonnation on how to interpret the tables that show estimated program 
impacts presented throughout this report). As shown in Table 1.2, Family Rewards reduced by 
nearly 1 1 percentage points the proportion of families with income at or below the poverty line, 
reduced by nearly 8 percentage points the proportion of families who reported difficulty making 
ends meet, and increased by 1 8 percentage points the proportion of families who reported that 
their financial situation was better than the previous year. 

Family Rewards decreased by about 2 percentage points the proportion of parents who 
had worked, according to unemployment insurance (UI) records. But it increased by nearly 6 
percentage points those who reported working, according to a survey that was conducted at 18 
months. No impacts were found on earnings from UI records. The discrepancy between these 
two assessments may suggest that there was a very small disincentive to work, at least in jobs 
captured by the UI system, because of the program’s additional income support, but also that 
some Family Rewards participants may have been additionally encouraged to do work that was 
not captured in the UI records (including self-employment). With regard to health outcomes, 
Family Rewards produced no effects on some outcomes, especially when base outcomes were 
very high, and had small to moderate effects on other outcomes. For example, Family Rewards 
increased by nearly 10 percentage points the proportion of parents who had two dental checkups 
and reduced by almost 3 percentage points the proportion of children who had a period with no 
health coverage. It also had positive impacts on parents’ self-rated health. 

Finally, Family Rewards did not improve school outcomes for elementary or middle 
school students related to school attendance rates or annual standardized test scores in math and 
English Language Arts during the first two years of the program. Data from the 18-month 
parent survey indicate that Family Rewards increased the likelihood that middle school students 
would be involved in school-related activities, such as programs to help with schoolwork or 
homework, school clubs, school musical programs, and dance or art lessons (not shown in 
Table 1.2). However, there is no indication so far that these extra efforts have translated into 
higher academic achievement for these children. 

For the oldest children, however, the impacts of the program are somewhat different 
than for their younger peers. Although the program had few effects on school outcomes for high 
school students overall, Family Rewards substantially improved the educational achievement of 
high school students who scored at or above the basic proficiency level on their eighth-grade 
standardized tests before random assignment. This group of better-prepared high school 
students saw almost a 6 percentage point reduction in the proportion of students who repeated 
the ninth grade, a 15 percentage point increase in the likelihood of having a 95 percent or better 



The Opportunity NYC Demonstration: Family Rewards 
Table 1.2 


Summary of Impacts for the Full Family Rewards Sample 



Program 

Control 

Difference 


Outcome 

Group 

Group 

(Impact) 

P-Value 

18 -month survey outcomes 





Selected impacts on material hardship and income 





Food security 3 (1 = low; 4 = high) 

Did not get needed medical care because of cost 

3.4 

3.2 

q 2 *** 

0.000 

in past 12 months b (%) 

6.5 

10.4 

_3 g *** 

0.000 

Family finances are not enough to make ends meet (%) 
Strongly or somewhat agree that the financial situation is 

34.1 

41.8 

_y g *** 

0.000 

better than last year (%) 

Percentage of families with household income at or below 

62.7 

44.5 

18.3 *** 

0.000 

c d 

the federal poverty level ’ (%) 

59.4 

70.0 

-10 7 *** 

0.000 

Sample size (total = 2,060) 

1,051 

1,009 



Selected impacts on employment and earnings 





Ever employed in UI records (%) 

56.2 

58.5 

_2 3 ** 

0.011 

Total earnings in UI records (S) 

12,114 

12,354 

-240 

0.284 

Currently employed at the time of the survey (%) 

59.9 

54.3 

5 6 *** 

0.000 

Sample size (total = 4,994) 

2,513 

2,481 



Selected impacts on health outcomes and job characteristics (%) 





Some or all of respondent's children had a period 





with no coverage 

14.6 

17.9 

-3.3 ** 

0.014 

Had a health checkup since random assignment 

93.2 

91.7 

1.5 

0.128 

Had at least two dental checkups since random assignment 

67.4 

57.9 

g 5 *** 

0.000 

Average self-rated health is excellent 

15.8 

13.5 

2.3 * 

0.064 

Currently being treated for any medical condition 

47.2 

44.4 

2.8 * 

0.092 

Sample size (total = 3,082) 

1,574 

1,508 



Education outcomes 





Impacts on attendance and test scores: 4th-graders (%) 





Attendance rate, Year 2 

87.9 

88.3 

-0.4 

0.689 

Attendance rate is 95% or higher, Year 2 

44.5 

41.6 

2.9 

0.225 

Percentage proficient on ELA test, Year 2 

67.6 

68.1 

-0.4 

0.828 

Percentage proficient on math test, Year 2 

80.3 

78.6 

1.7 

0.345 

Sample size (total = 1,726) 

862 

864 



Impacts on attendance and test scores: 7th-graders (%) 





Attendance rate, Year 2 

86.4 

87.6 

-1.2 

0.182 

Attendance rate is 95% or higher, Year 2 

36.6 

34.9 

1.6 

0.472 

Percentage proficient on ELA test, Year 2 

46.5 

46.0 

0.5 

0.824 

Percentage proficient on math test, Year 2 

61.9 

63.5 

-1.6 

0.414 

Sample size (total = 1,671) 

823 

848 




(continued) 
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Table 1.2 (continued) 



Program 

Control 

Difference 


Outcome 

Group 

Group 

(Impact) 

P-Value 

Impacts on attendance and test scores: 9th-graders (%) 





Enrolled in 10th grade 

70.3 

68.8 

1.5 

0.446 

Attendance rate is 95% or higher, Year 2 

28.8 

23.7 

5 9 *** 

0.006 

Earned at least 22 credits, Years 1 and 2 

45.0 

44.4 

0.6 

0.767 

Passed at least 2 Regents exams, Years 1 and 2 

38.2 

37.6 

0.7 

0.719 

Sample size (total = 1,979) 

988 

991 



9th-graders who scored at or above proficiency level on the 
annual math test in 8th grade 3 (%) 





Remained in 9th grade, Year 2 

3.0 

8.8 

g *** 

0.003 

Attendance rate is 95% or higher, Year 2 

51.1 

36.2 

14.9 *** 

0.000 

Earned at least 22 credits, Years 1 and 2 

72.7 

64.5 

8 i ** 

0.032 

Passed at least 2 Regents exams, Years 1 and 2 

77.6 

71.7 

5.9 * 

0.098 

Sample size (total = 584) 

298 

286 



9th-graders who scored below proficiency level on the annual 
math test in 8th grade e (%) 





Remained in 9th grade, Year 2 

22.1 

21.8 

0.3 

0.916 

Attendance rate is 95% or higher, Year 2 

21.8 

19.3 

2.5 

0.291 

Earned at least 22 credits, Years 1 and 2 

38.1 

40.1 

-2.0 

0.492 

Passed at least 2 Regents exams, Years 1 and 2 

22.9 

25.2 

-2.3 

0.343 

Sample size (total = 1,143) 

565 

578 




SOURCE: MDRC calculations from New York City Department of Education administrative records. 


NOTES: Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre-random assignment 
characteristics of family or sample members. 

Rounding may cause slight discrepancies in calculating sums and differences. 

Sample sizes may vary because of missing values. 

A two-tailed t-test was applied to the differences between outcomes for the program and control groups. 

The p- value indicates the likelihood that the difference between the program and control group arose by 
chance. 

Years 1 and 2 cover the 2007-2008 and 2008-2009 school years, respectively. 

ELA is the English Language Arts exam; UI is unemployment insurance. 

a The food security question describes food eaten by the family in the prior month: 1= Often not enough to eat; 
2 = Sometimes not enough to eat; 3 = Enough to eat but not always the kinds of food desired; 4 = Enough to eat of 
the kinds of food desired. 

b This excludes prescriptions. 

c Family Rewards payments are based on Seedco's Family Rewards data from Year 2 of the program, which 
include activities completed in September 2008 through August 2009. The monthly Family Rewards payment 
amount is calculated by dividing the annual reward amount by 12. The payment data do not include bonus 
payments that some families received for opening new bank accounts. 

d Annual household income was calculated by multiplying by 12 the respondent's income in the month prior to 
the survey interview. For program group members, it includes Family Rewards payments earned during Year 1 
(2007-2008) of the program. The federal poverty level was created based on annual income (monthly income 
multiplied by 12) and the household size at the time of the survey. The poverty threshold was measured according 
to the 2008 or 2009 Federal Poverty Guidelines, depending on when a respondent was interviewed. 

e In New York State, students who score at a level of 3 or higher on a 4-point scale are deemed "proficient.” 
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Box 1.1 

How to Read Impact Tables in This Report 


In the context of this evaluation, an “impact” is a measure of the effect of the Family 
Rewards program, represented by the difference between those teenagers and their parents 
randomly assigned to the program group, compared with those randomly assigned to the 
control group. All the tables in this report that show impacts use a similar format, with 
information provided for families in each of the program and control groups. Because 
families were assigned randomly either to the program group or to the control group, the 
effects of the program can be estimated by the difference in outcomes between the two 
groups. The “Difference” column in the table shows the differences between the two 
research groups’ outcomes — that is, the program’s estimated impacts on the outcomes. 
All impacts are calculated controlling for a range of pre -random-assignment characteris- 
tics to increase precision of the impact estimates. The p-value shows the probability that 
this difference, or impact, arose by chance. The number of asterisks indicates whether the 
impact is statistically significant at the 1 percent (***), 5 percent (**), or 10 percent (*) 
level. All estimates are regression-adjusted using ordinary least squares, controlling for 
pre-random-assignment characteristics of families or sample members. 


attendance rate (in Year 2), an 8 percentage point increase in the likelihood of staying on track 
for graduation over the first two years of the program, and an increase of about 6 percentage 
points in the likelihood of passing at least two Regents exams. By contrast, there are no statisti- 
cally significant impacts for the group of high school students who scored below proficiency on 
standardized tests. 

These findings represent preliminary results from the core evaluation of Family Re- 
wards. Results from the third to fifth year of follow-up are forthcoming. Because there were 
stronger positive impacts on educational outcomes for the ninth-graders relative to the other two 
age groups, the child and family embedded study focused on that age group. The embedded 
study sought to understand the effects of the program on changes in the family setting as well as 
a broader set of developmental processes and outcomes for teenagers that may have mediated 
the program’s impacts on educational outcomes or may have contributed to teenagers’ well- 
being more broadly two years into the program. 


The Family Rewards Child and Family Embedded Study: 
Theoretical Framework 

Family Rewards is built on the premise that financial incentives can benefit families in two 
ways: by reducing poverty in the short term by transferring cash to poor families, but also by 
encouraging behavioral changes among parents and children in ways that may benefit them in 
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the long run by making those cash payments contingent on the families’ participation in a set of 
key activities. These dual pathways are shown in Figure 1.1, which illustrates the hypothesized 
pathways of influence of the Family Rewards program. In short, each of the incentive offers (in 
the three domains of employment and training, education, and health) is thought to induce a 
small set of incentivized behaviors in the short term (for example, in parents’ participation in 
employment and training, in school attendance and attendance at parent-teacher conferences, 
and in doctor’s and dental visits and the maintenance of health insurance). This embedded study 
addresses the effects of the Family Rewards program on a set of key mediating processes and 
outcomes not assessed in the core evaluation that were thought to be key to understanding the 
impacts for the oldest children. 

Changes in these mediating processes in the short term were thought to lead to longer- 
term gains in outcomes of children, such as their academic achievement and health. Beneficial 
effects were also thought to occur as a result of the short-term increases in income and reduc- 
tions in poverty and hardship that families should experience by engaging in the incentivized 
behaviors and receiving rewards payments. Family Rewards also has the unique feature of 
providing cash payments for outcomes as well (for example, in children’s achievement test 
scores); as a result, changes in children’s academic achievement should further increase income 
impacts among families. All of these pathways of influence (on incentivized behaviors, family 
income, and key targeted outcomes for children) are being assessed as part of the core evalua- 
tion of the Family Rewards program. 

As discussed earlier, this report addresses the effects of Family Rewards on a number of 
additional mediating processes: (1) teenagers’ time use; (2) parents’ and teenagers’ spending 
and savings; (3) interactions between parents and teenagers; (4) teenagers’ approaches to 
schooling — their expectations, engagement, efficacy, and motivation; and (5) teenagers’ 
mental health and problem behaviors, key outcomes not directly targeted by the program. These 
areas are shown in bold in Figure 1 . 1 , as they are the focus of this report. The relations between 
these outcomes, the Family Rewards program, and those outcomes collected as part of the core 
evaluation are shown in Figure 1 . 1 and discussed further below. 

Time Use 

One of the key ways that comprehensive CCTs are intended to affect outcomes for the next 
generation is that, by offering incentives for educational outcomes, they change the daily 
activities in which teenagers engage. The idea was that by incentivizing educational outcomes, 
teenagers might spend more time in school-related activities, such as completing homework 
assignments or studying, and in achievement-oriented after-school activities, such as tutoring. 
These changes in time use were thought to affect not only their academic achievement, but also 
their engagement in problem behaviors with peers. 
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The Opportunity NYC Demonstration: Family Rewards 
Figure 1.1 

A Heuristic Model for the Child and Family Embedded Study 


CCT 

Intervention 


Offer of 
employment 
& training 
incentives 


Offer of 
- education 
incentives 


Offer of 
health 
incentives 


Incentivized 

Behaviors 


Mediating 

Processes 


Child 

Outcomes 



NOTES: All 3 child outcomes are hypothesized to be associated with one another, but lines indicating these associations are not shown. 
Bold boxes indicate mediating processes measured in the Family Rewards Child and Family Embedded Study. 





























Spending and Saving 

The Family Rewards intervention provided families with the offer of cash transfers that 
should result in a greater level of resources within families. Indeed, the interim results (dis- 
cussed above and in Riccio et al. 7 ) showed that the program successfully increased family 
income and reduced material hardship. The main goal of the incentives was to encourage 
parental investments in children’s education, by allowing parents to purchase tutoring services, 
after-school activities, and other academic resources for their children, thus supporting chil- 
dren’s academic achievement and long-term academic expectations. However, families might 
have decided instead to use the rewards to support basic necessities, on the one hand, or leisure 
and entertainment activities (such as movies) on the other. 

Less was hypothesized about how the program might affect teenagers’ own spending, 
although the hope was that the teenagers, like their parents, would invest in their future human 
capital. Did they spend additional resources on educational items? What about household items 
(food, clothing)? Or did they spend more on social activities? What does this tell us about how 
the increased resources might have benefited or harmed adolescents? 

Parent-Teenager Interactions 

The offer of incentives through Family Rewards was expected to affect interactions be- 
tween teenagers and their parents. More specifically, Family Rewards was expected to shape the 
climate of the family with regard to nomis and expectations about children’s participation in 
school and after-school activities and the value of school success and schoolwork. These 
changed norms and expectations might result in parents’ increased monitoring of their chil- 
dren’s activities. Notably, changes in parental expectations and pressure could also act as a 
double-edged sword and erode the potential positive effects that higher parental and family 
expectations and aspirations for their children’s academic success might have on children’s 
well-being. As such, these changed nonns might also increase conflict between parents and 
their adolescent children. 

Teenagers’ Approaches to School 

Family Rewards incentives — by offering educational incentives and through changes 
in family interactions — were also thought to change teenagers’ approach to schooling (that is, 
their academic expectations, school engagement, academic efficacy, and motivations). For 
example, it was thought that program-driven changes in parents’ encouragement of educational 


7 Riccio et al. (2010). 
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attainment (and parental socialization of such values) would shape children’s own perceptions 
of the value of educational achievement and engagement. By offering incentives for educational 
outcomes, it was thought that Family Rewards might encourage teenagers to be more interested 
in their learning and have higher expectations for their academic outcomes. These might all be 
reinforced if school attendance and greater time spent on schoolwork “paid off’ in temis of 
better academic outcomes. 

In the area of motivation, cash incentives have received considerable scrutiny, because 
of concerns that such rewards might potentially have negative effects on children’s motivation 
to learn. 8 However, it is not clear whether these findings would carry over to real-world field 
settings, especially where the challenges of living in impoverished communities might impinge 
on students’ levels of intrinsic motivation, resulting in lower levels of intrinsic motivation from 
the outset. 9 In fact, the designers of Family Rewards hoped that the offer of incentives would 
not discourage intrinsic motivation. 

Teenagers’ Mental Health and Problem Behaviors 

In addition to effects on the key targets of the program, research has rarely explored the 
effects of programs like Family Rewards on children’s mental health and problem behaviors. 
However, such a program might indeed have implications for such outcomes. For example, the 
greater income and resources generated by Family Rewards might well affect outcomes such as 
teenagers’ mental health and problem behaviors as well as the targeted outcomes of the pro- 
gram. 10 Moreover, effects on delinquency could be influenced by changes in teenagers’ time 
use, as more time spent on academics might reduce time spent on risk-taking behaviors. At the 
same time, negative effects could also occur if teenagers spent their money engaging in risky 
behaviors and/or if the program heightened parents’ expectations at the cost of teenagers’ 
mental health. 


Sample for the Current Study 

This report focuses on a subset of the children in the oldest age cohort (ninth grade at baseline) 
from the Family Rewards core study sample who were interviewed as part of the core evalua- 
tion follow-up at the 18-month time point (see Figure 1.2). As discussed earlier, nearly 4,800 
families were randomly assigned to the core evaluation. A little more than 3,000 of these 
families were selected and interviewed at the 18-month point for a follow-up survey as part of 


8 Deci (1971). 

9 Eccles and Wigfield (2002). 

10 Costello, Compton, Keeler, and Angold (2003). 
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The Opportunity NYC Demonstration: Family Rewards 
Figure 1.2 

Family Rewards Child and Family Embedded Study 
Sample Selection Process 



NOTE: Two teenager respondents are excluded from the Child Survey respondent sample because their parents 
did not complete the Parent Survey. 
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the core evaluation. A sample of 716 teenagers and their parents were selected from this group 
of families. Information was collected as part of this embedded child and family study on 5 1 1 of 
them, representing an effective response rate of 72 percent (77 percent of parents were contact- 
ed and gave consent for their children to participate; of these, 93 percent of children consented 
to participate and complete a survey). The embedded study sample was purposely selected to 
oversample teenagers with high levels of prebaseline math proficiency because of the larger 
impacts found on children who were proficient in math in the core study (to ensure a 50-50 split 
by baseline math proficiency). 1 11 

A telephone survey was administered to the 5 1 1 teenagers and their parents during the 
spring of their third year following random assignment (approximately 30 months after they 
entered the core study). The survey collected information from teenagers on their spending and 
saving patterns, time use, level of conflict in discussions of various topics with their parents, 
various educational processes, mental health, and risky behaviors. The parent survey collected 
information from parents on their spending and saving patterns, levels of monitoring their 
teenager, and level of conflict in discussions of various topics with their teenager. Sources, 
psychometric properties, and item-level detail for all measures collected on these surveys are 
provided in Appendix D. 

Table 1.3 shows the baseline characteristics of these 511 parents and teenagers. As can 
be seen in the table, the sample is comprised of largely single parents in households with 
incomes below 130 percent of the federal poverty line. A large proportion of families rely on 
public benefits, although about half were working full time. The parents in the sample are black 
and Hispanic; more than four- fifths are U.S. citizens. In terms of characteristics of high school 
students, nearly all attended public schools, and 13 percent were enrolled in special education. 
Only a little over a third of the students were found to meet learning standards for proficiency in 
two core subject areas: English Language Arts (ELA) and math. 

Analyses comparing the embedded study sample of parents and teenagers with the core 
study sample are presented in Appendix E. These analyses show that the two samples did not 
differ in important ways, with a few exceptions. Moreover, analyses comparing the results for 
educational outcomes of this embedded study sample show similarity between the findings 
from the core study and those in this sample, with limited impacts on educational outcomes for 
the full group of teenagers and positive impacts for those who scored at or above the basic 
proficiency level on their eighth-grade standardized math tests (although impacts on educational 

1 'in addition, since the core study sample was slightly unbalanced between program and control groups on 
parents’ baseline employment (see Riccio et al., 2010), stratification on parents’ baseline employment status 
was conducted to ensure better balance on this pre-random-assignment characteristic in the embedded study 

sample. Note that in both the core and embedded studies, impact analyses control for this baseline characteris- 
tic. 
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The Opportunity NYC Demonstration: Family Rewards 
Table 1.3 


Characteristics of Families at the Time of Random Assignment, 
by Research Group for the Embedded Study Sample 


Characteristic 

Program 

Group 

Control 

Group 

Family characteristics 

One-parent families (%) 

79.8 

83.1 

Average number of children in household 

2.4 

2.4 

Primary language spoken at home is English (%) 

78.6 

84.3 * 

Living in public housing (%) 

29.0 

33.9 

Receiving TANF or Safety Net Assistance (%) 

22.7 

22.3 

Receiving food stamps (%) 

60.0 

59.4 

At least one adult covered by public health insurance (%) 

68.0 

69.6 

Earnings above 130% of federal poverty level (%) 

16.3 

12.6 

Parent characteristics 

Female (%) 

95.8 

92.8 

Race/ethnicity (%) 

Hispanic/Latino 

45.0 

40.2 

Black, non-Hispanic/Latino 

53.8 

57.7 

Other 

1.2 

2.0 

Age (%) 

1 9-34 years 

13.7 

16.9 

35-44 years 

48.1 

53.4 

45-54 years 

30.5 

23.3 

55 years or more 

7.6 

6.4 

U.S. citizen (%) 

By birth 

68.7 

71.5 

By naturalization 

14.9 

15.3 

Legal permanent resident 

16.4 

13.3 

Education, highest degree or diploma earned (%) 

Less than high school diploma/GED 

37.5 

42.0 * 

High school diploma/GED 

20.7 

25.9 * 

More than high school diploma/GED 

41.8 

32.1 * 

Currently working full time (%) 

49.6 

49.0 


(continued) 
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Table 1.3 (continued) 


Characteristic 

Program 

Group 

Control 

Group 

Teenager characteristics 



Bom in the U.S. (%) 

89.7 

92.3 

Female (%) 

53.8 

54.0 

Average age (years) 

13.9 

14.0 

Race/ethnicity (%) 



Hispanic/Latino 

44.4 

37.7 

Black, non-Hispanic/Latino 

52.5 

59.4 

Other 

3.1 

2.9 

Type of school attended in the past year (%) 



Public or charter school 

99.2 

98.4 [ ] 

Private or parochial school 

0.8 

1.6 [ ] 

Enrolled in special education in the past year (%) 

13.4 

13.0 

Enrolled as English Language Learner in the past year (%) 

11.2 

11.7 

Meets English Language Arts learning standards (%) 

37.1 

33.6 

Meets math learning standards (%) 

40.5 

37.3 

Sample size (total = 511) 

262 

249 


SOURCE: MDRC calculations using data from Baseline Information Forms. 


NOTES: Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 

10 percent. 

In order to assess differences in characteristics across research groups, chi-square tests were used 
for categorical variables, and t-tests were used for continuous variables. 

Brackets ([ ]) around significance levels indicate that the chi-square tests for statistical 
significance are not valid because sample sizes within categories are too small. 

Rounding may cause slight discrepancies in calculating sums. 

Sample sizes may vary because of missing values. 

Public health insurance measures in this table exclude child information. 

outcomes are slightly larger for the embedded study sample of proficient children). Most 
importantly for the impact analyses presented in this report, there were very few differences 
between program and control group respondents in measured baseline characteristics for either 
parents or teenagers, indicating that the internal validity of the impact analyses was maintained 
in this subsample of the core study sample. 


Analyses for the Current Study 

Analyses presented in the current report focus on the differences between parents and teenagers 
assigned to the program group and those assigned to the control group in each of the areas of 
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interest discussed above, controlling for a small set of pre-random-assignment characteristics to 
increase precision. 12 As a result, analyses do not formally test the conceptual model presented in 
Figure 1.1. A better representation of the analytic framework for the results presented in this 
report is Figure 1.3. (Impacts on outcomes presented in the current report on the embedded 
study are shown in bold-lined boxes; those impacts presented in reports from the core study are 
shown in the lighter-lined boxes of the same figure, where each line represents an impact 
analysis that was conducted on a different set of outcomes.) What this figure highlights is that 
relationships between impacts are not tested in this report. For example, analyses are conducted 
to test whether the Family Rewards program produced differences between program and control 
groups in teenagers’ time use and whether the program also produced differences between 
program and control groups in teenagers’ problem behaviors; however, analyses are not 
conducted of whether those two impacts are related to each other. The design of this study 
allows for the strongest causal statements to be made about the first set of analyses, in that any 
differences between program and control groups can be confidently attributed to the Family 
Rewards program. Since the same confidence cannot be applied to relations between outcomes 
in the conceptual model, those are not the focus of this report. At the same time, the discussion 
does address the pattern of impacts across hypothesized mediators and outcomes as a means to 
begin to build a theory of change for the Family Rewards program that can be tested in future 
work. 


Organization of This Report 

This report presents findings on the impacts of Family Rewards on parents and their teenagers, 
well into their experience in the program (in the spring of their third school year in the pro- 
gram). Chapter 2 presents impacts on parents’ and teenagers’ spending as a result of the 
program, as well as on two aspects of family processes — teenagers’ time use and interactions 
between parents and teenagers. Chapter 3 reports impacts of Family Rewards on outcomes for 
the teenagers: their approaches to schooling (academic expectations, school engagement, 
academic efficacy, and motivation) as well as mental health and behavioral outcomes. Finally, 
Chapter 4 summarizes the findings and offers concluding thoughts about the implications of 
these findings for CCT interventions more broadly. 


12 A range of baseline covariates were included in all analyses. Covariates included average class size for 
grade level, ELA pretest score, math pretest score, school district, child gender, child race, special education 
status, the number of children in the household, whether English was spoken in the home, if the family was a 
single-parent or two-parent family, parents’ education, parents’ employment status, an indicator for if there was 
no mother in the home, and an indicator for whether random assignment occurred after September 2007. 
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The Opportunity NYC Demonstration: Family Rewards 
Figure 1.3 

Tested Impacts in the Core Study and the Child and Family Embedded Study 
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NOTES: All 3 child outcomes are hypothesized to be associated with one another, but lines indicating these associations are not shown. 
Bold boxes indicate mediating processes measured in the Family Rewards Child and Family Embedded Study 










Chapter 2 


The Impact of Family Rewards on Time Use, Spending 
and Saving, and Parent-Teenager Interactions 

This chapter presents impact findings of Family Rewards on teenagers’ time use, on parents and 
teenagers’ spending and savings, and on interactions between parents and teenagers. Impacts 
are presented for all parents and their teenage children in the embedded study sample. In 
addition, impacts are discussed by subgroups based on teenagers’ baseline proficiency at the 
point of random assignment, given that this baseline characteristic shaped program impacts on 
educational outcomes (see Chapter 1 and Riccio et al. 1 ). Results of impact analyses for boys and 
girls are also discussed briefly (the full set of such subgroup results are presented in Appendix 
B). 


As discussed in Chapter 1, Family Rewards was intended to change the way teenagers 
spent their time, with the goal of increasing the amount of time they spent on academic activi- 
ties. Findings show statistically significant program impacts on the proportion of teenagers in 
different time use groups based on the way they allocated their time, with Family Rewards 
reducing the proportion of teenagers who spent their time predominantly in social activities and 
increasing the proportion of teenagers who spent their time predominantly in academic activi- 
ties. This finding was especially true for the teenagers who were better prepared for school 
based on their test scores before random assignment. However, when examined activity by 
activity, there was no difference between the Family Rewards group and the control group in 
the average amount of time teenagers spent participating in academic activities. 

The designers of Family Rewards also thought that increased resources might lead to 
increased parental investment in activities (such as tutoring) and materials (such as books) that 
might promote learning. What was far less clear, however, is whether parents would also use 
the resources to support basic necessities and leisure activities, and, indeed, how teenagers 
might spend their own earned rewards. In either case, how parents and teenagers spent their 
resources may provide some insight into the extent to which the program might have encour- 
aged families to invest in teenagers’ human capital in ways that could have the longer-term 
effects on reducing intergenerational poverty, a central goal of conditional cash transfer (CCT) 
programs. Indeed, the findings discussed in this chapter indicate that Family Rewards did 
increase the amount of money parents spent on their teenagers (with higher levels of spending 
by program group parents than control group parents on academic activities and supplies as well 
as basic necessities) and had a positive impact on parents’ saving for their teenager’s future 


'Riccio et al. (2010). 
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education. Teens in the Family Rewards group spent somewhat less on necessities and leisure 
activities than control group teens, perhaps because parents were spending more. It is important 
to remember that the embedded survey was conducted in late winter and spring, before the time 
when teenagers could earn large rewards (for example, for test scores or Regents exams). A 
survey fielded in the summer could have yielded somewhat different results on spending. 

Finally, Family Rewards was expected to change the way parents and their teenagers 
interacted with each other, with the hope that parents might monitor more closely their teenag- 
er’s activities with the introduction of incentives for their child’s education. At the same time, 
there was a concern that the introduction of such incentives might increase parent-teenager 
conflict, especially regarding the earning and spending of reward payments. In fact, however, 
neither the benefits to parental monitoring nor the increases in conflict as a result of Family 
Rewards were observed. 

These three sets of results, on teenagers’ time use, spending and savings, and parent- 
teenager interactions, are discussed in more detail in this chapter. These findings set the stage 
for an exploration of Family Rewards’ effects on teenagers’ academic processes and other 
outcomes that are presented in Chapter 3. 


Time Use 

By incentivizing particular behaviors (for example, school attendance) and the achievement of 
longer-term outcomes (for example, reading and math achievement scores), Family Rewards 
was intended to change the daily activities in which the teenagers engaged. For example, it was 
hoped that children would spend more time in school-related activities, such as completing 
homework assigmnents and studying, and in achievement-oriented after-school activities, such 
as tutoring, because the outcomes of such behaviors (that is, test scores) are directly incentivized 
by the program. Teenagers’ daily activities also might be shaped by changes in parental em- 
ployment and labor force participation, as children might assume more household and adult-like 
responsibilities, such as caring for younger siblings, cleaning, shopping, or cooking, when their 
parents worked. Alternatively, it is possible that teenagers who earned their own money would 
not need to work for pay as much and thus could focus their efforts on other activities. 

As such, this embedded study was designed to examine the impacts of the program on 
the way teenagers spent their time. To assess this, teenagers responded to a series of questions 
asking how they spent their time between 2 p.m. and bedtime on a typical weekday. Using these 
data, two approaches were used to understand the impacts of Family Rewards on teenagers’ time 
use: (1) The first approach assessed time use by assigning teenagers into “time use groups” by 
relying on a statistical approach called cluster analysis (see Appendix C for technical details). 
This approach considered all activities simultaneously, with groups of teenagers characterized by 
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distinct “patterns” or profiles of time use based on the way they allocated their time (for example, 
one group might be characterized by particularly high amounts of time on one activity, say 
participation in sports, and low amounts on the other activities; another might be characterized by 
moderate levels of involvement in many activities). One advantage of this approach is that it 
reflects that the time spent on each of the activities is dependent on others, such that an increased 
proportion of time spent in one activity would directly lead to a decreased proportion of time 
spent in other activities. With this approach, impacts of Family Rewards are examined on the 
proportion of teenagers in each of the groups. 2 (2) The second approach assessed time use in a 
more typical fashion, one activity at a time. In this latter case, the amount of time each teenager 
spent in each activity was computed based on start and end times for each activity, and impacts 
are examined on the average amount of time spent in such activities. 

Notably, both approaches provide unbiased estimates of program impacts but are some- 
what different conceptualizations of the same information. In the first case, the analysis will 
provide information about whether there is an impact on the way teenagers allocated their time 
across activities; in the second case, the analysis will provide infonnation about whether there is 
an impact on the average amount of time teenagers spent in each activity. 

Impacts on Time Use Groups 

As discussed above, time use was first assessed using an approach that identified groups 
of teenagers based on their patterning of activity participation in all activities. This analysis 
identified four groups of teenagers (see Box 2.1): (1) those who spent their time predominantly in 
academically oriented activities (homework, after-school activities); (2) those who spent their 
time predominantly in socially oriented activities (hanging out with friends or family, talking on 
the phone or texting); (3) those who spent a large proportion of time watching TV and on the 
computer; and (4) those who spent the most time in maintenance activities (eating, napping, 
grooming, commuting) and working (doing chores, work for pay). 

Figure 2. 1 shows the proportion of time spent in each activity for each of these identified 
time use groups. While individuals within a group will not have identical patterns of time use, 
their patterns will be more similar to each other than to those of teenagers in the other groups. 3 


2 Notably, unlike subgroups, which are based on baseline characteristics of families pre-random- 
assignment, these “groups’’ are based on postrandomization patterns, and therefore impacts are examined on 
the proportion of teenagers in each of the groups. 

3 Notably, there is heterogeneity in teenagers’ reported time use even within the “time use groups.” But 
teenagers are classified in one group over another because their activity scores are more similar to other 
teenagers classified in that group than to those in another group. For example, two teenagers in the socially 
oriented group will have similar but not identical profiles: One teenager in the socially oriented group may 
spend 69 percent of his time engaged in social activities, 16 percent in academic activities, 2 percent in 

(continued) 
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Box 2.1 


Time Use Groups 

Teenagers were divided into groups derived from an analysis of the proportion of time 
they spent on each of their reported activities from 2 p.m. until they went to bed on a 
typical weekday. Activities were initially categorized into academics, work, religious 
activities, social activities, maintenance activities (eating, napping, grooming, commut- 
ing), family chores, and time spent on the computer or watching TV for this analysis. Then 
groups of teenagers were identified based on an approach called cluster analysis. Cluster 
analysis identifies groups of individuals with similar values across a set of variables. This 
analysis identified four groups of teenagers: 

Academically oriented. On average, this group spent over half of their time engaged in 
academic activities (50.7 percent), and the rest of their time engaged in TV/computer time 
(14 percent), maintenance activities (20.7 percent), and social activities (6.2 percent). 

Socially oriented. On average, this group spent over half of their time engaged in social 
activities (that is, hanging out with friends/family, on the phone; 52.6 percent). They spent 
most of the rest of their time engaged in TV/computer time (15.2 percent), maintenance 
activities (19 percent), and academics (6.6 percent). 

TV/computer oriented. On average, this group spent more than half of their time engaged 
in watching TV or on the computer (56.7 percent). They spent most of the rest of their 
time on maintenance activities (17 percent), academics (14.6 percent), and social activities 
(4.8 percent). 

Maintenance/work oriented. On average, this group spent the most time in maintenance 
activities (40.7 percent) and significantly more time than any others working for pay and 
doing family chores (nearly 7 percent versus less than 2 percent for working, and about 6 
percent versus less than 3 percent for chores). A large part of the rest of their time was 
equally split between academic activities (10.6 percent), TV/computer time (12.1 percent), 
and nonacademic organized activities (14.4 percent). 


The academically oriented teenagers reported, on average, spending half their time in academic 
activities, less than 10 percent of their time in social activities, and about 15 to 20 percent of their 
time watching TV/on the computer or in maintenance activities (such as eating, napping, 
grooming, commuting). By contrast, the socially oriented teenagers spent 


TV/computer time, and the remaining 13 percent in maintenance activities, while another teenager in the group 
may spend 38 percent of his time on social activities, 15 percent on academics, 16 percent on TV/computer 
activities, and the remaining 30 percent on maintenance activities. While these two teens do not have identical 
time use profiles, the patterns in the proportion of time they spend in the various activities are similar. 
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SOURCE: MDRC calculations from Family Rewards 30-Month Embedded Study Survey. 


half their time on social activities, less than 10 percent on academic activities, and, like the 
academically oriented group, about 15 to 20 percent of their time watching TV/on the 
computer or in maintenance activities. The TV/computer teens spent half their time watching 
TV/on the computer and less than 10 percent of their time on all other activities (including 
educational and social), with the exception of maintenance activities. Finally, the mainte- 
nance/work group spent about 40 percent of their time on maintenance activities. While the 
proportion of their total time spent working and doing chores (both at about 6 percent of their 
time) did not represent a majority of their time, they did spend a much higher proportion of 
their time on these activities than any of the other groups (which all were working less than 2 
percent of the time and doing chores less than 3 percent). Importantly, the groupings are quite 
robust to how certain activities are categorized in the analysis (see Appendix C), including, 
for example, removing time spent on computers from the TV group, since it is difficult to 
distinguish whether such time is an educational, social, or leisure activity. And, as also 
discussed in Appendix C, the groups are associated with outcomes in expected ways (that is, 
the academically oriented group has better academic outcomes than the other three groups; 
the socially oriented group has higher levels of substance use than the other three groups). 

The impact of the Family Rewards program was then examined on the proportion of 
teenagers in these “time use groups.” Results are shown in Figure 2.2. 4 As shown in the 
figures, about 40 percent of teenagers in the control group were in the maintenance/work time 
use group, a quarter in the TV/computer time use group, a little more than 20 percent in the 
social group, and 16 percent in the academically oriented group. Family Rewards reduced the 
proportion of socially oriented teenagers and increased the proportion of academically oriented 
teenagers. More specifically, teens in the program group were more likely to be categorized in 
the academically oriented group compared with the socially oriented group (24 percent of 
teenagers compared with 1 5 percent), a pattern that was not observed in the control group. 

This same analysis was repeated by teenagers’ baseline math proficiency, given that 
this baseline characteristic shaped program impacts on their educational outcomes (see 
Chapter 1 and Riccio et al. 5 ). Results are presented in Figures 2.3 and 2.4. As shown in the 
figures, the results observed above for the entire sample are concentrated among those 
teenagers who were rated as proficient on the math test at the end of eighth grade. More 
specifically, in the nonproficient subgroup, the patterns of time use across the program and 
control groups are not different from each other. In the proficient subgroup, however, there 
are significant program and control group differences. Pair-wise comparisons show that 


4 These results were from a multinomial logit analysis with treatment status as an independent variable 
(along with baseline covariates) and the four groupings as the dependent variable that showed a significant 
treatment effect on membership in the outcome. Post-hoc comparisons between groups revealed the 
significant differences reported above. 

5 Riccio et al. (2010). 
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Impacts on Teenagers' Membership in Time Use Groups 
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NOTE: Estimates were regression-adjusted using ordinary least squares, controlling for pre-random assignment characteristics of family 
and sample members. 
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program group teenagers in the proficient group are more likely to be in the academically 
oriented group versus the socially oriented group compared with controls, and also are more 
likely to be in the academically oriented group versus the maintenance/worker group compared 
with controls. These impacts are larger than those in the full embedded study sample shown 
above. 


Differences were also explored by gender (see Appendix C). The results observed for 
the entire sample were concentrated in girls only. More specifically, program group girls are 
more likely to be in the academically oriented group versus the socially oriented group com- 
pared with controls and also are more likely to be in the academically oriented group versus the 
maintenance/worker group compared with controls. These impacts are larger than those in the 
full embedded study sample shown above. No differences were found among program and 
control groups for boys. 

Impacts on Individual Activities of Time Use 

To complement the above approach with an approach more typically used in impact 
analysis, program impacts were also assessed by examining the average amount of time spent in 
each kind of activity for the program and control groups. Unlike the previous approach, this 
approach considers each activity independently. Teenagers’ reported time use was divided into 
three “kinds” of activities (productive, maintenance, and leisure activities; see Box 2.2). Impacts 
on these categorized measures of time use are shown in Table 2.1. The length of time spent in 
each of the three different categories of activities was examined. 

There are no statistically significant impacts on teenagers’ reports of the way they spent 
their time on weekday afternoons and evenings when activities are examined individually. 
When asked about their activities from the after-school hours (2 p.m.) until bedtime, teenagers 
reported spending on average about two and a half hours a day on productive activities (educa- 
tional, work, or religious), about two hours on maintenance activities (napping, eating, groom- 
ing, commuting) and the most time — three and a half hours — on leisure activities (with 
friends, watching TV). The amount of time spent on each of these categories was quite similar 
for teenagers in the program and the control groups. 

Table 2.1 also presents data on teenagers’ reported time spent with different people 
(parents, friends) during their participation in the above-mentioned activities. As shown in the 
table, teenagers spent the most time with their families (about four hours) and, secondly, with 
peers (about two and a half hours). Family Rewards increased the amount of time teenagers 
reported spending with their family members by three-fourths of an hour; teenagers in the 
program group reported spending more time with their family (4.6 hours) than teenagers in the 
control group (3.9 hours) on a typical weekday. There are no statistically significant differences 
in the time teenagers spent alone, with peers, with coworkers, or with teachers for the full 
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Box 2.2 


Time Use Measures of Time Spent in Activities 

Teenagers reported on what they did, whom they were with, and where they were on a 
typical weekday beginning at 2 p.m., until they went to bed. Activities were categorized as 
follows: 

Productive activities. Time spent working, in school-related activities (for example, in 
school, in academic lessons or activities), in religious activities, in unstructured activities, 
and in arts and cultural activities. 

• Productive educational. Time spent in class/school, reading, school- 
work/studying, academic lessons or activities, and clubs or nonacademic activi- 
ties/lessons 

• Productive work. Time spent working for pay 

• Productive religious. Time spent in prayer/worship/meditating/attending church 

Maintenance activities. Time spent on eating, personal care, commuting, chores and 
errands, and resting/napping 

Leisure activities. Time spent socializing, watching TV, on the computer, or shopping 


sample (although the impact on peers is in the negative direction, consistent with the analyses 
showing a reduced proportion of socially oriented teenagers in the Family Rewards group). 

In addition to the impacts for all teenagers, impacts on average time spent in activities 
were examined by teenagers’ baseline test proficiency (see Table 2.2). There are no statistically 
significant differences between proficiency groups in the amount of time spent in individual 
activities. There are, however, some differences in impacts with regard to the individuals 
teenagers spent their time with, by baseline proficiency groups. Proficient teenagers in the 
program group spent less time alone than their counterparts in the control group (1.1 versus 1.8 
hours on a typical weekday), while there are no significant differences between nonproficient 
program and control group teenagers. In addition, proficient teenagers in the program group 
were less likely to spend time with coworkers (0 hours versus 0.2 hours), while there are no 
significant program impacts among the nonproficient teenagers. 

Impacts on time use were examined for boys and girls, and few differences were found 
(see Appendix Table B. 1). 
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Table 2.1 

Impacts on Hours Spent in Weekday Activities 


Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P-value 

Kind of activity 

Productive 

2.4 

2.4 

0.0 

0.841 

Productive educational 

1.6 

1.5 

0.1 

0.678 

Productive work 

0.2 

0.2 

0.0 

0.922 

Productive religious 

0.0 

0.0 

0.0 

0.561 

Maintenance 

2.3 

2.1 

0.2 

0.286 

Leisure 

3.5 

3.5 

0.0 

0.970 

Accompanying individual during activity 

Alone 

1.3 

1.5 

-0.2 

0.228 

Family 

4.6 

3.9 

0 g *** 

0.008 

Peers 

2.3 

2.6 

-0.4 

0.123 

Coworkers 

0.2 

0.2 

0.0 

0.771 

Teacher 

0.8 

0.9 

-0.1 

0.318 

Sample size (total = 511) 

262 

249 




SOURCE: MDRC calculations from the Family Rewards 30-Month Embedded Study Survey. 

NOTES: Statistical significance levels indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 
percent. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre-random 
assignment characteristics of family or sample members. 

Rounding may cause slight discrepancies in calculating sums and differences. 

Sample sizes vary because of missing values. 

Teenagers were asked about a more discrete set of categories for a weekend day. There 
are no statistically significant differences between teenagers in the program and control groups 
in the amount of time spent in any of these activities (results shown in Appendix C). The single 
exception is that teenagers reported spending less time working for pay on the weekend in the 
Family Rewards group compared with the control group (0.2 hours for program group teenagers 
compared with 0.4 hours for control group teenagers). 

In sum, while there are statistically significant program impacts on the proportion of 
teenagers in the varying time use groups, there are no statistically significant impacts on the 
amount of time spent in each activity across all teenagers. However, Family Rewards was found 
to significantly increase the proportion of time teenagers spent with family, which is quite 
consistent with the results on time use groups discussed above. These findings are interpreted to 
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Table 2.2 
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mean that (1) Family Rewards has an impact on how teenagers pattern their time use, not on 
each discrete activity of time use per se, and (2) these impacts are observed for the proficient 
teenagers only. The findings for the proficient teenagers are consistent with the findings from 
the core report that found impacts on academic outcomes only for this group. 6 

Despite the divergence in findings, it is critical to note that both approaches represent 
valid estimates of the program’s impact, although each answers a slightly different question. 
Given that, is there an explanation for these seemingly inconsistent findings between the two 
approaches to examining time use? Several explanations are possible. First, it may be that there 
was an increase in the amount of time spent on academic activities, but that impact was too 
small to detect when examined over the entire sample. Indeed, if the impact that was found on 
time spent in educational activities — 0.1 of an hour, or 6 minutes (see Table 2.1) — was 
confined to only the 5 percent of the sample of teenagers who shifted from a socially oriented 
group to an academically oriented group, it would represent 2 hours of increased time for this 
smaller group. Thus, one explanation may be that there is a substantial shift in amount of time 
spent in activities but that the group affected is too small to observe those impacts when 
examined over the full sample. Second, it may be that the shift in time use that did occur was 
not so much related to total time spent in academically oriented activities per se, but to the 
patterning of activity participation, or the way they allocated their time across activities. That is, 
those children who shifted from the socially oriented group to the academically oriented group 
may have had small shifts in their time spent in several of the activities. These shifts may have 
been such that they “fit” better with the academically oriented group instead of the socially 
oriented group, even if the total time spent in academics did not increase substantially relative to 
what it would have been in the control group. Finally, it may be that the impact on the propor- 
tion of teenagers in time use groups is a statistical fluke, and only with replication of these 
findings in other studies of CCTs can that be determined. Future work will examine each of 
these possible explanations in greater detail. 

However, one way to build confidence in these results is to examine the impacts of Fami- 
ly Rewards on other outcomes that might be related to time use. Doing so can determine if there is 
at least a consistent “story” of positive program impacts across outcome domains that can lend 
greater credibility to the findings presented here. The fact that Family Rewards had positive 
impacts on academic outcomes for the more proficient teenagers in ways that “line up” with the 
findings reported here on time use groups is noteworthy in this regard. It will also be important to 
note whether impacts are observed for other outcomes examined later in this chapter and in the 
next chapter as part of this embedded study that can lend credibility to these findings. 


6 Riccio et al. (2010). 
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Spending and Saving 

Family Rewards has an explicit antipoverty component that should result in a greater level of 
resources within families and, as a result, greater spending or saving within families. However, 
whether or not those increased resources would be spent in ways that would benefit children’s 
long-term human capital was an open question. On the one hand, parents might use the in- 
creased resources to support their children’s academically enriching experiences or to save for 
future educational opportunities, consistent with the goals of the CCT program. But parents 
might also decide that these resources needed to be spent on basic necessities (such as food and 
clothing) or to support leisure activities (such as movies or eating out) that can be difficult to 
afford for low-income families. In other welfare and antipoverty programs, parents whose 
income increased spent their money on necessities for their children (child care, food, and 
clothing). 7 However, parents might be more inclined to spend their resources on their children in 
the case of Family Rewards, since children “earn” some of the rewards through their own 
perfonnance on tests or attendance in school. 

Given that rewards for certain educational incentives were awarded in part to children, 
the second question is how teenagers themselves would spend the increased resources they had 
access to under the program. This information was not collected in the core study; by fielding a 
survey to these older children themselves, this embedded study provided an opportunity to 
examine this question directly. 

In interpreting these findings, it is critical to be cognizant of the timing of the survey 
fielding relative to the delivery of reward payments to parents and teens. The interviews took 
place in the late winter and spring of their third school year in the program (about 30 months 
following random assignment). During this period, teenagers could be receiving attendance 
rewards, and parents could be receiving rewards for health-related visits and employment. 
However, rewards for test scores, which carry some of the highest payment amounts and are 
paid directly to the teenage children, were not distributed until later in the summer when the test 
results were released. Therefore, these data provide only a “snapshot” of this particular period 
during the year of reward receipt and may not reflect how teenagers spent some of the largest 
reward payment they received. Furthermore, as information was gathered only on certain areas 
of spending, these data cannot provide a complete accounting of families’ full budgets. 

Box 2.3 describes the information on spending that was gathered and how it was cate- 
gorized. Findings on the impact of Family Rewards are presented in Table 2.3. 

As shown in Table 2.3, parents reported spending on average more than $600 on their 
children in a one-month period in these areas (which notably exclude other areas such as hous- 

7 Michalopoulos, Bos, Lalonde, and Verma (2000). 
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Box 2.3 


Measures of Parents’ and Teenagers’ Spending 

Parents and teenagers were asked about spending on a number of items that were thought 
to comprise the majority of their spending. Spending was asked about in a way that 
program and control group families could answer and, therefore, reflects total dollars 
spent (irrespective of the source of those dollars), rather than spending from the rewards 
payments in particular. Participants were asked about their spending using various time 
periods that encouraged recall of precise information (the previous month for spending on 
food and clothing that happens frequently; school-year spending for items that are typical- 
ly purchased in those longer intervals; based on work by Lugo-Gil and Yoshikawa, 2006) 
and then converted to average monthly spending for all items to “add up” items in com- 
mon sets of categories. 

Spending was categorized into three areas: productive spending (that is, on educational 
and other structured activities and materials), maintenance spending (that is, on basic 
necessities), and leisure spending (that is, “fun” money). 

Parents’ Spending 

Productive spending. Spending on books, school-related costs, such as uniforms, out-of- 
school activities, and games or sporting goods 

Maintenance spending. Spending on food, health care (such as prescriptions or co- 
payments), and clothes or shoes 

Leisure spending. Spending on computer, entertainment, DVDs, CDs, or video games 

Teenagers’ Spending 

Productive spending. Spending on school supplies, books or magazines, and after-school 
activities 

Maintenance spending. Spending on household groceries or cleaning supplies, spending 
on food and clothes 

Leisure spending. Spending on music, movies/movie rentals or video games, electronics, 
and alcohol or cigarettes 


ing costs), with approximately two-thirds spent on maintenance items (basic necessities such as 
food and clothing). Parents reported spending about $100 per month on leisure activities, such 
as entertainment, and a little more than that on productive activities for their teens (such as 
books and school activities and expenses). 
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Table 2.3 


Impacts on Parents' and Teenagers' Spending and Saving 


Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P-Value 

Parent report 

Total spending in 1 month' 1 ($) 

Total spending 

695 

624 

71 * 

0.056 

Productive spending 

156 

125 

31 *** 

0.007 

Maintenance spending 

434 

418 

16 

0.552 

Leisure spending 

109 

82 

27 *** 

0.008 

Saving and giving 

Save for child's future education (%) 

54.1 

41.5 

12.6 *** 

0.005 

How often do you set aside money? 

(1 = never, 5 = every week) 

2.1 

1.8 

0.3 ** 

0.025 

How much money do you set aside? ($) 

124 

71 

54 

0.207 

Teenaser report 

Total spending in 1 month b ($) 

Total spending 

230 

301 

_72 ** 

0.019 

Productive spending 

34 

38 

-4 

0.600 

Maintenance spending 

162 

213 

-51 ** 

0.020 

Leisure spending 

35 

51 

-16 * 

0.070 

Saving and giving 

Save money (%) 

75.8 

70.1 

5.8 

0.156 

How much did you save? ($) 

89 

73 

17 

0.395 

Give money as gift, loan, donation (%) 

41.9 

42.6 

-0.7 

0.870 

Sample size (total = 511) 

262 

249 




SOURCE: MDRC calculations from the Family Rewards 30-Month Embedded Study Survey. 

NOTES: Statistical significance levels indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 
percent. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre-random 
assignment characteristics of family or sample members. 

Rounding may cause slight discrepancies in calculating sums and differences. 

Sample sizes may vary because of missing values. 

A two-tailed t-test was applied to the differences between outcomes for the program and control 
groups. 

The p-value indicates the likelihood that the difference between the program and control group 
arose by chance. 

“Spending on different items was reported for varying time periods ranging from 1 to 6 months. All 
numbers reported in this table are scaled to a 1 -month interval. 

b Spending on different items was specifically asked about "in the past 4 weeks." 
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Not surprisingly, given families’ increased financial resources, Family Rewards in- 
creased the amount of money parents spent on their children. Parents in the program group, 
relative to those in the control group, reported spending $71 more overall on their teenagers 
over a one-month period. Levels of spending were higher in the productive category (by $3 1 per 
month), which includes spending on books, school-related costs, and games and sporting goods. 
They were also higher in the leisure category (by $27 per month), which includes spending on 
entertainment and DVDs, CDs, or video games. Statistically significant impacts on overall 
spending for maintenance activities are not found. Examining individual items shows signifi- 
cant positive impacts on spending on school activities and games/sporting goods within the 
larger productive category and on entertainment in the larger leisure category. 

As shown in the second panel of impacts, 41.5 percent of parents in the control group 
reported saving for their teenager’s future education, a relatively high proportion of families. 
Family Rewards has a statistically significant positive impact on this outcome. Parents in the 
Family Rewards group were about 13 percentage points more likely to be saving for their 
teenager’s future education than parents in the control group (54. 1 percent compared with 41.5 
percent). They also reported saving more frequently (although the average of 2 on the 5-point 
scale indicates relatively infrequent saving for most families in both groups). However, parents 
in the program group did not report putting more money aside than their control group counter- 
parts, on average. 

These overall impacts on savings are strong and statistically significant for parents of 
teenagers who were proficient in math at baseline, but not for the parents of the nonproficient 
teenagers (see Appendix Table A.l). More specifically, parents of proficient children were 25 
percentage points more likely to report saving for their teenager’s future educational expenses, 
while there are no statistically significant impacts for the nonproficient teens. However, impacts 
on spending are not statistically significantly different across the two groups. With regard to 
gender subgroups, impacts on savings are much larger and statistically significant for girls and 
not for boys (see Appendix Table B.2). 

As shown in the bottom panel of the table, while there were positive impacts of Family 
Rewards on parents’ reported spending, there were somewhat unexpected reductions in chil- 
dren’s reported spending as a result of Family Rewards. More specifically, teenagers in the 
program group reported spending less money overall ($72 per month) than their counterparts in 
the control group, with that reduction concentrated in the maintenance ($51 per month) and 
leisure ($16 per month) spending categories. Further analyses revealed that this may have been 
a substitution effect — in which parents’ increased spending as a result of the program meant 
that teenagers did not have to contribute their own resources to family-related expenses, such as 
food and clothing. The fact that the reduction is concentrated in the maintenance and leisure 
categories is consistent with this interpretation. At the same time, there were no significant 
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impacts on teenagers saving or donating money, which would be consistent with this substitu- 
tion hypothesis. 

The fact that there are not higher levels of spending or saving among program group 
members compared with control group members may be because the interviews took place 
when the rewards teenagers themselves received were quite modest (for example, attendance 
rewards of $25 per month paid directly to teenagers were available during the period of the 
interviews but not the larger rewards for credits earned and test score results). The lower levels 
of spending for program group teenagers, despite these modest rewards, may have been the 
result of program group members’ lower likelihood of working for pay (see discussion earlier of 
time use) and the greater support coming from parents’ own contribution to these areas of 
spending. That said, it was unexpected that this reduction in spending was not accompanied by 
an increase in savings. 


Parent-Teenager Interactions 

Family Rewards was thought to change parent-teenager interactions in a number of ways. On 
the one hand, rewards might shape the climate of the family regarding norms and expectations 
about children’s participation in school and after-school activities, and, as a result of such 
changes, parents might spend more time being involved in and monitoring their children’s 
activities in and out of school. At the same time, designers expressed concern that Family 
Rewards, by introducing incentives to families, might increase parent-teenager conflict over 
time use and money during an already tumultuous period in families’ lives. To address these 
questions, infomiation was collected on measures of parents’ monitoring and teenagers’ 
disclosure of their whereabouts and activities as well as parent-teen conflict (see Box 2.4). 
These findings are presented in Table 2.4. 

Parents were asked about how frequently they monitored their teenager’s activities and 
how much they knew about where their teenager was and with whom. In a parallel fashion, 
children were asked how much they disclosed this same kind of information to their parents. 
Neither of these measures showed any impacts of the Family Rewards program, although 
average rates were relatively high for both parents and teenagers (approximately 4 on a 5-point 
scale), with values on parental monitoring slightly higher than those on teenagers’ disclosure. 

Teenagers were asked about 13 different topics of potential conflict and, for each, the 
intensity of the conversation between them and their parents. Impacts were examined on the 
intensity of the interaction for all topics combined as well as for money-related and school- 
related topics, in particular, given the focus of the Family Rewards program. As shown in Table 
2.4, parents and teens reported low to moderate levels of conflict across the topics (an average 
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Box 2.4 


Measures of Parent-Teenager Interactions 

Parental monitoring reflects parents’ tracking and surveillance of their child’s behavior 
and activities. Examples of items include: “Do you know what your child does during 
his/her free time?” “Do you know what your child spends his/her money on?” and “In the 
last month, has there been a time when you had no idea where your child was at night?” 

Teenager disclosure reflects the extent to which teenagers tell their parents about their 
behaviors and activities. Examples of items include: “Do you talk about how you are 
doing in the different subjects in school?” “Do you keep much of what you do in your free 
time secret from your primary caregiver?” and “Do you hide a lot from your primary 
caregiver about what you do at night/on weekends?” 

Family conflict refers to the number of items parents and teenager discuss and the intensi- 
ty level of these discussions (that is, angry versus calm). Eligh conflict refers to more 
angry discussions. 


score of 2 on a 5 -point scale for both teenagers and then parents, where 1 is calm and 5 is 
angry), with somewhat higher levels of conflict for parents’ reports of money-related issues. 
There are no statistically significant program impacts on any of these measures. While not 
shown in the table, parents did report discussing more money-related issues with their children 
(2.55 versus 2.37 out of 3 possible money-related issues asked about), as expected due to the 
offer of rewards, but those discussions were not more intensively negative or conflictual. 

Impacts were examined on both of sets of outcomes for the two key subgroups exam- 
ined throughout this chapter: by teenagers’ baseline proficiency and by gender. In both cases, 
there are no statistically significant differences across these subgroups in impacts on either of 
these outcomes of parent-teenager interaction (see Appendix Tables A.2 and B.3). In all cases, 
there are no patterns of statistically significant impacts of the Family Rewards program. 

The next chapter turns from impacts on family processes to impacts for the teenagers 
themselves, focusing on their approaches to schooling, a key possible mechanism for any 
educational impacts, and their mental health and problem behaviors, key outcomes that might 
be indirectly affected by the Family Rewards program. 
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Table 2.4 


Impacts on Parent-Teenager Interactions 


Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P- Value 

Monitoring/disclosure 





Parental monitoring (5 = high monitoring) 

4.3 

4.3 

0.0 

0.928 

Teenager disclosure to parent (5 = high disclosure) 

3.9 

3.8 

0.1 

0.495 

Intensity level of family conflict (5 = angry, 1 = calm) 





Parent report 





Discussion of all issues 

2.0 

2.1 

0.0 

0.832 

Discussion of money-related issues 

1.9 

1.9 

0.0 

0.971 

Discussion of school-related issues 

2.4 

2.3 

0.1 

0.499 

Teenager report 





Discussion of all issues 

2.0 

2.0 

-0.1 

0.296 

Discussion of money-related issues 

1.8 

1.9 

-0.1 

0.520 

Discussion of school-related issues 

2.1 

2.2 

-0.2 

0.104 

Sample size (total = 511) 

262 

249 




SOURCE: MDRC calculations from the Family Rewards 30-Month Embedded Study Survey. 

NOTES: Statistical significance levels indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 
percent. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre-random 
assignment characteristics of family or sample members. 

Rounding may cause slight discrepancies in calculating sums and differences. 

Sample sizes may vary because of missing values. 

A two-tailed t-test was applied to the differences between outcomes for the program and control 
groups. 

The p-value indicates the likelihood that the difference between the program and control group 
arose by chance. 
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Chapter 3 


The Impact of Family Rewards on Teenagers’ 
Approaches to School and Mental Health and Problem 

Behaviors 


Chapter 3 presents impact findings from this embedded study on teenagers’ approaches to 
schooling and problem behaviors. Specifically, these analyses provide information on the effect 
of the Family Rewards program on academic processes, as well as on outcomes not directly 
targeted by the Family Rewards program: mental health outcomes such as depression and 
anxiety and problem behaviors such as aggression and delinquency and substance use. 

A central premise of the Family Rewards program was that by offering educational in- 
centives for attendance and achievement outcomes, increased school attendance and achieve- 
ment would in turn increase children’s interest in school and schoolwork and their sense of 
academic efficacy. At the same time, designers worried about undermining intrinsic motivation 
by providing external rewards for academic outcomes. The findings discussed in this chapter 
show that the program had neither the beneficial effects that were hoped for in terms of academ- 
ic engagement, efficacy, or expectations nor any harmful effects on teenagers’ motivation. 

Also discussed in this chapter are the impacts of Family Rewards on outcomes not di- 
rectly targeted by the program. Arguably, this is one of the most important contributions of this 
embedded study, as it informs understanding of the full effects on outcomes for teenagers of this 
holistic CCT program. Here the focus is on teenagers’ mental health — where designers were 
worried about undue pressure on children as a result of the program’s incentives — and prob- 
lem behaviors such as aggression and substance use — where it was less clear whether the 
program’s impacts might be positive or negative. Here the results are unambiguously positive: 
On outcomes where there were concerns about negative effects (mental health), Family Re- 
wards had no impacts; on outcomes where the expected direction of the impact was not clear 
(aggression and substance use), there are clear benefits of the program in reducing problem 
behaviors. While these latter findings require replication, as earlier studies of CCTs did not 
examine such outcomes, they suggest potentially broader benefits of the Family Rewards 
program. 


Teenagers’ Approaches to Schooling 

The designers of the Family Rewards program were hopeful that program-driven changes in 
parents’ perceived value of educational attainment and parental socialization of such values 
might shape their children’s own perceptions of the value of educational achievement. By 
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consequence, Family Rewards could influence teenagers’ academic processes, such as choices 
and decisions about engaging in and persisting in school and schoolwork. Similarly, Family 
Rewards could heighten teenagers’ own expectations about their long-term educational goals. 
Changes in such important academic processes might also encourage teenagers to work harder 
in school and score higher on tests, both potentially critical for long-term educational success. 

At the same time, the use of cash incentives in the area of children’s academic 
achievement has been the topic of much controversy, particularly in younger children, because 
of its potential effects on children’s motivation to leam. A long history of lab-based research has 
suggested that the provision of financial rewards can undermine intrinsic motivation. 1 In short, 
while extrinsic rewards may increase education-relevant behaviors in the short term, they may 
reduce intrinsic motivation for learning later, when the extrinsic rewards are withdrawn. In fact, 
more recent work 2 has suggested a more nuanced view, such that if financial incentives can 
promote individuals’ sense of competence, they can actually support intrinsic motivation. 
Notably, this research is also based primarily on lab experiments, leaving us with little infor- 
mation about how these processes may operate in real-world settings and, more specifically, for 
low-income parents and their children receiving a range of incentives in a holistic CCT pro- 
gram. 


Given that data were collected for this embedded study while teenagers and their fami- 
lies were still receiving rewards, how might impacts on motivation inform our understanding of 
the program’s long-term consequences? If Family Rewards increased children’s intrinsic 
motivation to leam during this period (or at least did not undermine intrinsic motivation), the 
short-term benefits of the program on children’s education behaviors would be more likely to 
endure after the program ended. But if the program decreased children’s intrinsic motivation, 
any short-term benefits of the incentives might wash out later when the incentives were with- 
drawn. 


Measures of teenagers’ approaches to schooling are shown in Box 3.1, and findings are 
presented in Table 3.1. 

In the first panel of Table 3.1, findings are presented regarding teenagers’ academic ex- 
pectations. Despite parents’ greater savings for their children’s future education, there are no 
statistically significant program impacts on teenagers’ aspirations and expectations to attend and 
complete college, perhaps because rates were very high (nearly 91 percent of children expected 
to attend college and almost 86 percent expected to finish college). Such high rates of expecta- 
tions regarding college attendance are consistent with findings from other research on adoles- 


'Deci (1971). 

2 Deci and Ryan (2000). 
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Box 3.1 


Measures of Teenagers’ Approaches to Schooling 

School engagement reflects engagement in classroom and school activities. High en- 
gagement refers to trying hard to do well, paying attention in class, working hard in 
school, and listening carefully in class. 

Academic efficacy reflects students’ perceptions of their academic abilities and ability to 
succeed at challenging academic material. 

Motivation reflects the driving force to learn. Three measures of teenagers’ motivation 
were examined in this study: 

• Motivation to learn refers to whether one is extrinsically or intrinsically moti- 
vated to learn; that is, does the student perform academic tasks because of exter- 
nal reinforcements (for example, “I will get in trouble if I don’t do it”; “I may get 
a reward if I do it”) or because of an internal drive (for example, “It’s important 
for me to do well in school.”). Larger positive scores indicate intr ins ic motiva- 
tion, while larger negative scores indicate extrinsic motivation. 

• Mastery goal orientation refers to the wish to become proficient in a topic to the 
best of one’s ability. 

• Performance-avoidant goal orientation refers to behaving in a particular way 
in order to avoid an undesirable outcome (for example, avoiding the demonstra- 
tion of incompetence to others). 


cent expectations. 3 But research has found a weak relationship between such high expectations 
and actual college attendance rates for low-income adolescents. 4 In reality, college attendance 
and graduation rates are quite low; only one in three low-income adolescents actually attends 
college, and only one in seven actually finishes college. 5 

In the second panel of the table, findings are presented on teenagers’ reports about the 
extent to which they were engaged in school, with items assessing the extent to which they were 
paying attention in class and working hard. No statistically significant impacts are found on this 
measure of school engagement. Taken in the context of the findings on attendance reported in 
Chapter 1 and on time use reported in Chapter 2, it appears that while teenagers were 


3 Kao and Thompson (2003). 

4 Rouse (2004). 

5 Bedsworth, Colby, and Doctor (2006). 
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Table 3.1 

Impacts on Teenagers' Approaches to Schooling 


Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P- Value 

Academic expectations (%) 

Hope to complete some college 

89.8 

91.7 

-2.0 

0.431 

Think will complete some college 

90.5 

91.8 

-1.3 

0.612 

Hope to finish college 

84.9 

85.5 

-0.5 

0.860 

Think will finish college 

85.8 

84.3 

1.6 

0.608 

School processes 

School behavioral engagement 

4.3 

4.3 

0.0 

0.557 

Academic efficacy 

2.9 

2.9 

0.1 

0.252 

Motivation 

Motivation to learn (-7 = extrinsic, 7 = intrinsic) 

0.7 

0.5 

0.2 

0.421 

Mastery goal orientation 

4.5 

4.5 

0.0 

0.789 

Performance-avoidance goal orientation 

3.1 

2.9 

0.2 

0.146 

Sample size (total = 511) 

262 

249 




SOURCE: MDRC calculations from the Family Rewards 30-Month Embedded Study Survey. 

NOTES: Statistical significance levels indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 
percent. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre -random 
assignment characteristics of family or sample members. 

Rounding may cause slight discrepancies in calculating sums and differences. 

Sample sizes may vary because of missing values. 

A two-tailed t-test was applied to the differences between outcomes for the program and 
control groups. 

The p-value indicates the likelihood that the difference between the program and control group 
arose by chance. 


more likely to be attending school and more likely to be “academically oriented” in terms of 
how they were spending their time after 2 p.m. (more time on schoolwork), teens in the program 
group were not more interested or listening more in school than their control group peers while 
they were in the classroom. In addition, there are no statistically significant impacts on teens’ 
sense of their own “efficacy” in their academic abilities or their thinking that they could succeed 
in handling challenging material in the classroom. This is unfortunate, since these processes 
have been linked in educational research to success in school. Perhaps the lack of effects on 
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these academic processes explains the relatively modest benefits on educational outcomes of the 
Family Rewards program. It may be that the provision of rewards needs to be paired with 
interventions that change the quality of educational experiences to affect the way teenagers 
approach schooling. 

The third panel of Table 3.1 shows impacts of Family Rewards on motivational factors. 
Teenagers were asked about their extent of extrinsic versus intrinsic motivation: There are no 
statistically significant impacts of the Family Rewards program on this measure. There are also 
no statistically significant impacts of the Family Rewards program on measures of teenagers’ 
mastery-goal orientation (their motivation to become proficient in a topic) or their performance- 
avoidant goal orientation (their motivation to do their schoolwork to avoid a particular out- 
come). The fact that the program did not undermine intrinsic motivation should be viewed quite 
favorably, especially considering the concern about how the provision of rewards might 
undermine motivation among already struggling students. At the same time, the fact that 
intrinsic motivation and teenagers’ motivation to become proficient was not increased is in line 
with the above-mentioned findings on teenagers’ engagement and efficacy. Family Rewards did 
not appear to substantially change the way teens approached schooling, except with regard to 
how they spent their time (that is, attending school and becoming more academically oriented in 
how they spent their after-school hours). 

Given the greater impacts on educational outcomes (especially test scores) for those 
teenagers who were better prepared academically for school based on baseline test scores, it 
might have been expected that Family Rewards would have positive impacts on these measures 
for those teenagers who scored at proficiency levels in eighth grade. These results are shown in 
Appendix Table A.3. The findings show that there were no differences in program impacts by 
baseline proficiency, with few effects for either group. There were also no differences in 
impacts on these measures for boys and girls (see Appendix B.4). 


Teenagers’ Mental Health and Problem Behaviors 

In addition to effects on the key targets of the program, this study was intended to address the 
possible effects of programs like Family Rewards on teenagers’ mental health and problem 
behaviors. Understanding the effects on these nontargeted outcomes is critical to understanding 
the broader effects of the Family Rewards program. The addition of a survey of the teenagers 
themselves as part of this embedded study was necessary to reliably gather this information, 
since parents are especially poor reporters of teenagers’ behaviors in these areas. 


49 



A number of studies have found an association between family income and children’s 
mental health outcomes. 6 Therefore, the fact that Family Rewards increases income within 
families bodes well for potential reductions in mental health problems. In addition, the literature 
has found a strong link between academic competence and psychosocial outcomes among 
teenagers, such that the greater educational benefits of the Family Rewards program could 
increase self-esteem and reduce mental health and problem behaviors. 7 At the same time, 
parents in the program group, in an effort to receive the financial incentive rewards, might 
heighten their expectations for their children’s academic achievement and increase parental 
pressure, at the cost of teenagers’ mental health. While the designers of Family Rewards were 
aware of this concern and attempted to set rewards at levels to avoid such unintended conse- 
quences for children, there was little time to test empirically these assumptions before the 
program began. Therefore, it was important to do so as part of this embedded study. 

With regard to problem behaviors, it was similarly unclear whether Family Rewards 
might have positive or negative effects. On the one hand, the message behind the program is 
clearly about the value of schoolwork, which might encourage teenagers to spend more time on 
their homework and other school-related activities rather than spend time engaging in risky 
behaviors. Yet, with greater resources in hand (in particular because some of the educational 
rewards were given directly to teenagers), older children might spend their money on the kinds 
of activities that would increase their engagement in problem behaviors (such as drinking and 
using drugs), to the detriment of their longer-term outcomes. 

For these reasons, this embedded study included key measures of mental health and 
problem behaviors as reported by the teenagers themselves (described in Box 3.2). Findings on 
the impact of Family Rewards on these measures are shown in Table 3.2. 

As shown in the top panel of Table 3.2, there are no statistically significant impacts of 
the Family Rewards program on measures of teenagers’ depression and anxiety. Given concerns 
that parents would put undue pressure on their children and increase their children’s anxiety, 
this “null” finding should be seen as good news for the program. 

The second panel presents results on problem behaviors. While there are no statistically 
significant impacts on items reflecting delinquent activity (such as vandalism and theft), 
teenagers in the Family Rewards program did report lower levels of aggression with peers 
compared with their control group counterparts (a 10 percentage point reduction). Moreover, 
there were benefits of the Family Rewards program on reducing teenagers’ substance use, as 
discussed in detail below. 


6 Costello, Compton, Keeler, and Angold (2003); Gershoff, Aber, and Raver (2005). 

7 Roeser, Eccles, and Strobel (1998); Roeser, Eccles, and Freedman-Doan (1999). 
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Box 3.2 


Measures of Teenagers’ Mental Health and Problem Behaviors 

Depression and Anxiety reflect a teenager’s level of each construct on an index from 0- 
13. Teenagers answered 13 yes/no questions regarding depression, and 13 yes/no ques- 
tions regarding anxiety; scores were summed. Examples for depression include: “Nothing 
is fun anymore”; “I feel sad.” Examples for anxiety included: “I worry about little things”; 
“I get nervous.” 

Delinquent behaviors reflect acts that would be considered a criminal offense if commit- 
ted by an adult. Items include: “I took something from someone at school that did not 
belong to me”; “1 broke or ruined something on purpose that belonged to the school.” 

Aggressive behaviors reflect acts intended to cause harm to others. Items include: 
“Teased students to make them angry”; “Threatened to hurt or hit someone.” 

Any substance use was assessed with four items about whether or not teenagers smoked 
cigarettes, drank alcohol, used marijuana, and used hard drugs during the past month. 

Peer substance use was assessed with three items about how many friends drank alcohol, 
used marijuana, and used hard drugs in the past month. A four-point scale was used for all 
items, with 1 = “none”; 2 = “a few”; 3 = “some”; and 4 = “most.” 


With regard to teenagers’ own use of substances, the rates shown probably reflect an 
underreporting of use, with only about 7 percent of control group teenagers reporting smoking 
cigarettes, nearly 27 percent reporting using alcohol, nearly 14 percent reporting using marijua- 
na, and less than 1 percent reporting using hard drugs. Family Rewards resulted in statistically 
significant reductions in teenagers’ use of substances, for a number of key substances. Statisti- 
cally significant impacts are shown in the percentage of teenagers’ own use of alcohol (an 
almost 14 percentage point reduction) and marijuana (a 7 percentage point reduction). No 
statistically significant impacts are found for the already very low rates of use of cigarettes or 
hard drugs (although the frequency of cigarette use was significantly reduced in the program 
group, not shown in table). Qualitative analysis of program group participants reveals that there 
was some misunderstanding among families that Family Rewards was a program that rewarded 
“good behavior” generally. 8 This misunderstanding might have led some teenagers to engage in 
less risky behaviors; some might have found it easier to reduce their problem behaviors than 
increase their academic achievement. 


8 Greenberg, Dechausay, and Fraker (2011). 
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The Opportunity NYC Demonstration: Family Rewards 

Table 3.2 


Impacts on Teenagers' Mental Health and Problem Behaviors 


Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P- Value 

Effect 

Size 

Mental health outcomes 

Depression (0 to 13 scale) 

1.6 

1.5 

0.1 

0.557 

0.05 

Anxiety (0 to 13 scale) 

3.6 

4.0 

-0.4 

0.148 

-0.13 

Problem behaviors 

Delinquent behaviors (%) 

46.1 

52.7 

-6.6 

0.141 

-0.13 

Aggressive behaviors (%) 

14.1 

24.1 

-9.9 *** 

0.004 

-0.25 

Any substance use in past month (%) 

17.5 

32.6 

-15.2 *** 

0.000 

-0.35 

Ever smoked cigarettes (%) 

3.8 

6.5 

-2.7 

0.167 

-0.12 

Ever drank alcohol (%) 

12.9 

26.6 

-13.6 *** 

0.000 

-0.34 

Ever used marijuana (%) 

6.6 

13.5 

-6.9 ** 

0.010 

-0.23 

Ever used hard drugs (%) 

0.4 

0.4 

-0.1 

0.911 

-0.01 

Number of friends using substances in past month 

( 1 = none, 2 = a few, 3 = some, 4 = most) 

1.81 

1.99 

-0.19 *** 

0.004 

-0.27 

Number of friends drinking alcohol 

2.19 

2.40 

-0.21 ** 

0.022 

-0.21 

Number of friends using marijuana 

2.07 

2.34 

-0.27 *** 

0.007 

-0.25 

Number of friends using hard drugs 

1.18 

1.24 

-0.05 

0.286 

-0.10 

Sample size (total = 511) 

262 

249 





SOURCE: MDRC calculations from the Family Rewards 30-Month Embedded Study Survey. 

NOTES: Statistical significance levels indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre-random assignment 
characteristics of family or sample members. 

Rounding may cause slight discrepancies in calculating sums and differences. 

Sample sizes may vary because of missing values. 

A two-tailed t-test was applied to the differences between outcomes for the program and control groups. 

The p-value indicates the likelihood that the difference between the program and control group arose by chance. 


One skeptical interpretation of these results is that teenagers in the program group be- 
lieved that their behavior, including their use of alcohol and drugs, would affect their program 
rewards. Therefore, they would be more likely than their control group counterparts to report 
lower levels of such use (even though they might be using such substances as frequently). In 
this case, the lower rates would be simply a result of a reporting bias, not a “true” benefit of the 
program on teens’ behavior. To investigate this and to examine the impact of Family Rewards 
on peer influences in their own right, teenagers were also asked about their peers ’ behavior, 
which may not be subject to the same kind of reporting bias. In the case of these outcomes, 
Family Rewards was also found to show benefits. Rates for teenagers’ peers are (not surprising- 
ly) higher than teens own reported use, at approximately 80 percent, 72 percent, and 1 8 percent 
for any friends using alcohol, marijuana, and other drugs, respectively (data not shown). But as 
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seen in the bottom panel of Table 3.2, teenagers in the program group reported statistically 
significantly fewer friends who used alcohol and marijuana than those in the control group 
(when asked how many friends used these substances on a scale ranging from none to most). 
These findings show a consistent pattern of reduction in risky behaviors for the teenagers in the 
Family Rewards program. This is important, given that these are outcomes that were not 
directly targeted by the program. Notably, these effects occurred despite the greater resources to 
which these teenagers had access in the program group, suggesting that the teenagers were not 
using their increased resources to purchase drugs or alcohol. 

These findings are consistent with effects on time use reported in Chapter 2. The pro- 
gram appears to have changed the way teenagers spent their time (that is, there were fewer 
socially oriented teens), and perhaps as a result, these teenagers were behaving in less aggres- 
sive ways with their peers and using substances less. It is probable that the program was not 
directly affecting teenagers’ friends’ behaviors. Rather, the program might have been affecting 
the choice of the peer groups with whom teens were spending their time. As the program began 
just as the teenagers entered high school, it might have affected the development of peer groups 
at this important school (and developmental) transition. 

Impacts again were examined by teenagers’ proficiency status at baseline as well as by 
gender. Despite the greater educational benefits for the more proficient teenagers, effects on 
problem behaviors were similar across these subgroups. Moreover, effects on substance use did 
not vary substantially for boys and girls. Results of both of these subgroups are presented in 
Appendix Tables A.4 and B.5. 

In the next and final chapter, the findings presented in this chapter — neutral effects on 
academic processes and mental health outcomes and reductions in problem behaviors — are 
discussed in the context of the other findings of the Family Rewards program. Also, the impli- 
cations of these findings are discussed for the potential long-term effects of this program on 
outcomes for teenagers as they move into adulthood. 
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Chapter 4 

Conclusion 


Chapter 4 summarizes the key findings of this report on the child and family embedded study of 
the Family Rewards program. The findings are placed in the global context of holistic condi- 
tional cash transfer (CCT) policies that target multiple domains, and the chapter discusses 
possible next steps for this area of research and intervention. 


Key Findings 

The evidence emerging from this embedded study is encouraging, showing that some behaviors 
of parents and children can be changed for the good by the CCT incentives offered by the 
Family Rewards program. And in cases where concerns were raised that the program could 
cause increased parent-teenager conflict, increased anxiety on the part of teenagers, and de- 
creased intrinsic motivation for teenagers, there was no evidence that the program had any of 
these unintended consequences. At the same time, a number of plausible processes that could be 
activated as a result of the program were not affected, highlighting areas that might be directly 
targeted in future interventions that include CCTs. 

Five main conclusions emerge from this study of the Family Rewards program: 

• Family Rewards increased the proportion of teenagers who were aca- 
demically oriented in how they spent their time. Family Rewards did lead 
to a shift in the way teenagers allocated their time. While there were no over- 
all differences in the average hours they spent engaged in different types of 
activities (for example, school-related activities, leisure activities, self-care 
activities), teenagers in the program group were more likely to be categorized 
as being in an academically oriented group and less likely to be categorized 
as being in a socially oriented group than the teenagers in the control group, 
based on the proportion of their time spent in the after-school hours. This 
finding was concentrated among the teenagers who were academically profi- 
cient based on their baseline test scores. This is consistent with findings from 
the core study that these teenagers improved their educational outcomes as a 
result of the program. 

• With regard to the way teenagers and their parents spent their money, 
Family Rewards had modest positive effects in facilitating parents’ in- 
vestment in children’s human capital. Parents were more likely to save for 
their child’s future educational expenses and were more likely to invest fi- 
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nancial resources in their teenagers (for example, spending on books, maga- 
zines, and school activities) as a result of the increase in income. As a result, 
it appears that teenagers did not have to contribute their own resources to 
these family-related expenses as extensively and thus reduced their spending 
relative to their control group peers. 

• Family Rewards did not affect interactions between parents and teenag- 
ers, for better or worse. On the one hand, concerns about greater parent- 
child conflict were not borne out in the analysis of the effects of the Family 
Rewards program. But parents were also not found to engage in greater mon- 
itoring of their teenager’s behavior, as might be expected if Family Rewards 
was causing large changes in how families interact with their teenagers. 

• Family Reward did not affect the academic processes thought to mediate 
any benefits to student academic outcomes. While students were more 
“academically oriented” in terms of how they spent their after-school time, 
they did not report greater engagement in school (such as listening in class or 
paying attention) or a greater sense of efficacy about schoolwork. The time 
use group findings suggest that they spent more time on schoolwork, even 
though they were not more interested in their learning while in school. On 
the positive side, the provision of rewards did not appear to undermine intrin- 
sic motivation, a central concern about this program at the outset. 

• Family Rewards reduced teenagers’ problem behaviors, key outcomes 
not directly targeted by the intervention. Teenagers in the program group 
reported lower levels of aggression and lower levels of substance use (alco- 
hol and marijuana use) than their peers in the control group, which could 
bode well for their long-tenn outcomes. 


Putting the Findings in Context: Research on CCTs 

Comprehensive CCTs have been tested in a number of international contexts: in nine Latin 
American countries and at least 10 others, primarily in South Asia and Africa. Rigorous 
evaluation studies of CCTs in Mexico, and early reviews of work in Latin America” and other 
parts of the world, 3 have documented improvements in child and adult health, 4 school attend- 


'Gertler (2004); Skoufias (2005). 
2 Rawlings (2005). 

3 Adato and Bassett (2008). 
4 Rivera et al. (2004). 
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ance, 5 and, when the transfers are sufficiently high, household poverty and food adequacy. 
Probably the most influential of these models is Mexico’s Oportunidades program. Launched in 
1997, this program was designed to promote preventive health activities, nutrition, and school 
attendance among children. Evaluated with an experimental design, Oportunidades was found 
to improve nutritional status for young children and to increase enrollment rates at the second- 
ary school level by small amounts. 6 7 8 Yet, the Oportunidades program generally did not affect 

7 

older children’s health status or scores on achievement tests. 

Recently, a comprehensive review of the design, evaluation, and impact of CCTs in 19 
different nations conducted by economists at the World Bank found reductions in poverty and 

o 

more use of health and education services.' However, when it came to what they referred to as 
“final outcomes,” such as child learning and child health status in the long term, mixed impacts 
were found across the range of CCT initiatives. This inconsistency in key developmental 
outcomes suggests that the effects of CCTs on their intended aim to improve human capital 
formation in the long run have yet to be fully identified. Understanding how and why CCTs 
may be (or may not be) affecting children’s mental health, behavior, and academic motivation is 
a key contribution of this study to the literature and, moreover, is critical to understanding what 
policy levers may be associated with improving developmental outcomes for at-risk children. 


Looking Forward: The Next CCT and Future Findings from Family 
Rewards 

As discussed in detail in Riccio et al., 9 the Family Rewards program had modest impacts on 
high school students’ academic achievement, but only for a portion of the sample (those 
children who scored at proficiency levels on baseline test scores). The results of this embedded 
study shed light on some hypotheses to explain why this may have occured. There were 
changes in the way families allocated financial resources and small changes in the amount of 
time students spent on academic activities rather than social activities, but only for the academi- 
cally proficient teenagers and their parents. There were no statistically significant impacts on 
key hypothesized mediators among families, such as parental monitoring. Perhaps even more 
importantly, key student-level academic processes such as perceived academic efficacy as well 
as academic expectations were unchanged by the program. Perhaps, then, the program did not 
change enough of the key mediating mechanisms, and as a result, the expected changes in final 
outcomes were not observed across a larger proportion of the sample. 

5 Schultz (2001). 

6 Schultz (2004). 

7 Behrman, Parker, and Todd (2005); Behrman, Sengupta, and Todd (2000). 

8 Fiszbein and Schady (2009). 

9 Riccio et al. (2010). 
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These results suggest several directions for future work. First, more attention needs to 
be paid to redesigning incentives that are effective among teenagers who are not academically 
proficient. Second, future CCT initiatives in the United States might consider redesigning 
incentives to directly target these mediating processes in order to produce larger and more 
sustained impacts on outcomes. For example, rather than incentivizing academic achievement 
directly, rewards could offer incentives for students to attend academic tutoring programs or to 
join academic clubs, which have been shown to improve academic achievement and might 
change the way students approach schooling more effectively. Moreover, CCT programs 
might be more effective at changing academic processes when paired with programs to improve 
the quality of the educational experiences of children. Students in the Family Rewards program 
were attending a mixed group of schools in terms of quality; some of the schools were of 
relatively low quality. Perhaps the impacts on attendance and in the way students spent their 
after-school time were not sufficient to change how students approached schooling in the 
context of the type of school they attended. 

ft will be critical to examine the effects of the Family Rewards program in the long 
tenn. While Family Rewards did not have many of the hoped-for effects on academic processes 
in terms of the way students approached schooling, it did have positive impacts on students’ 
activity participation (in tenns of more teens being in an academically oriented group) and on 
parents’ saving for children’s future education. Combined with the reductions in teenagers’ 
aggression and substance use, these outcomes point to potential long-term benefits of the 
program for these teenagers as they move into adulthood. Fortunately, results from both the core 
study and the embedded study are being reviewed for implications for the redesign of CCT 2.0 
through the new efforts supported by the Social Innovation Fund. 

Future reports on this study will present findings on the program’s impacts at later 
points in the program. One will examine impacts three years after random assigmnent, around 
the time that the program itself was ending, and the final report will examine results over a five- 
year follow-up period, including two years after the program ended. The postprogram follow-up 
period will allow the evaluation to determine the long-term implications of the Family Rewards 
program and those of the particular findings from this embedded study of teenagers and their 
parents. 


10 Coie and Krehbiel (1984). 


58 



Appendix A 

Supplementary Tables for Chapters 2 and 3: Proficiency 

Subgroup Findings 




The Opportunity NYC Demonstration: Family Rewards 
Table A.l 


Impacts on Parents' and Teenagers' Spending and Saving, by Proficiency Level on 8th-Grade Math Test 



Scored at or above proficiency level 
on annual math test in 8th grade 

Scored below proficiency level 
on annual math test in 8th grade 



Program 

Control 

Difference 

Program 

Control 

Difference 

Sig. 

Outcome 

Group 

Group 

(Impact) 

Group 

Group 

(Impact) 

Parent renort 

Total spending in 1 month 3 ($) 

Total spending 

724 

601 

124 ** 

684 

630 

54 


Productive spending 

154 

126 

28 

158 

123 

35 ** 


Leisure spending 

104 

73 

31 ** 

113 

87 

26 * 


Maintenance spending 

470 

403 

67 

415 

421 

-6 


Saving and giving 

Save for child's future education (%) 

63.3 

38.3 

25.0 *** 

47.9 

43.4 

4.4 

tt 

How often do you set aside money? 

(1 = never, 5 = every week) 

2.3 

1.8 

0.5 ** 

2.0 

1.8 

0.1 


How much money do you set aside? (S) 

99 

101 

-2 

143 

49 

95 


Anyone help pay expenses for child? (%) 

32.3 

21.2 

11.1 * 

0.3 

0.2 

0.1 


How much help was provided this year? ($) 

134 

87 

47 

226 

114 

112 


Teenager renort 

Total spending in 1 month b ($) 

Total spending 

175 

249 

.74 ** 

266 

333 

-67 


Productive spending 

34 

30 

4 

36 

39 

-3 


Maintenance spending 

117 

179 

-62 *** 

190 

238 

-48 


Leisure spending 

24 

40 

-16 

41 

57 

-16 


Saving and giving 

Save money (%) 

74.1 

66.1 

8.1 

77.5 

72.1 

5.4 


How much did you save? ($) 

67 

63 

4 

102 

80 

22 


Give money as gift, loan, donation? (%) 

39.1 

47.9 

- 8.8 

44.1 

39.2 

4.9 


How much did you give? ($) 

8 

22 

_13 *** 

20 

15 

5 

tt 

Sample size (total = 511) 

106 

93 


156 

156 




(continued) 



Table A.l (continued) 


SOURCE: MDRC calculations from the Family Rewards 30-Month Embedded Study Survey. 

NOTES: Statistical significance levels indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Differences across subgroup impacts were tested for statistical significance. Statistical significance levels are indicated as follows: 
ttt = 1 percent; || = 5 percent; f = 10 percent. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre-random assignment characteristics of family or sample 
members. 

Rounding may cause slight discrepancies in calculating sums and differences. 

Sample sizes may vary because of missing values. 

A two-tailed t-test was applied to the differences between outcomes for the program and control groups. 

“Spending on different items was reported for varying time periods ranging from 1 to 6 months. All numbers reported in this table are scaled to a 
one-month interval. 

b Spending on different items was specifically asked about "in the past 4 weeks." 



The Opportunity NYC Demonstration: Family Rewards 
Table A.2 


Impacts on Parent-Teenager Interactions, by Proficiency Level on 8th-Grade Math Test 



Scored at or above proficiency level 
on annual math test in 8th grade 

Scored below proficiency level 
on annual math test in 8th grade 


Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

Sig. 

Monitorine/disclosure 

Parental monitoring (5 = high monitoring) 

4.5 

4.4 

0.1 

4.3 

4.3 

0.0 


Teenager disclosure (5 = high disclosure) 

4.0 

4.0 

0.0 

3.8 

3.7 

0.0 


Intensity level of family conflict 15 = anurv. 1 = calm) 

Parent report 

Discussion of all issues discussed 

1.9 

1.9 

0.0 

2.1 

2.2 

0.0 


Discussion of money-related issues 

1.7 

1.7 

0.0 

1.9 

2.0 

-0.1 


Discussion of school-related issues 

2.1 

2.1 

0.0 

2.5 

2.5 

0.1 


Teenager report 

Discussion of all issues discussed 

1.9 

2.1 

-0.2 * 

2.0 

2.0 

0.0 


Discussion of money-related issues 

1.6 

2.0 

_0 4 ** 

1.9 

1.8 

0.1 

tt 

Discussion of school-related issues 

2.0 

2.3 

-0.3 * 

2.1 

2.2 

-0.1 


Sample size (total = 511) 

106 

93 


156 

156 




SOURCE: MDRC calculations from the Family Rewards 30-Month Embedded Study Survey. 

NOTES: Statistical significance levels indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Differences across subgroup impacts were tested for statistical significance. Statistical significance levels are indicated as follows: ttt = 1 percent; 
tt = 5 percent; f = 10 percent. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre -random assignment characteristics of family or sample 
members. 

Rounding may cause slight discrepancies in calculating sums and differences. 

Sample sizes may vary because of missing values. 

A two-tailed t-test was applied to the differences between outcomes for the program and control groups. 



The Opportunity NYC Demonstration: Family Rewards 
Table A.3 

Impacts on Teenagers' Approaches to Schooling, by Proficiency Level on 8th-Grade Math Test 



Scored at or above proficiency level 
on annual math test in 8th grade 

Scored below proficiency level 
on annual math test in 8th grade 


Program 

Control 

Difference 

Program 

Control 

Difference 
(Impact) 1 ®' 

Outcome 

Group 

Group 

(Impact) 

Group 

Group 

Academic expectations (%) 

Hope to complete some college 

96.5 

96.4 

0.1 

84.9 

89.2 

-4.3 

Think will complete some college 

97.0 

95.8 

1.2 

85.4 

89.9 

-4.5 

Hope to finish college 

92.7 

92.1 

0.6 

79.4 

81.8 

-2.4 

Think will finish college 

95.3 

92.2 

3.1 

78.9 

79.8 

-0.8 

School processes 

School behavioral engagement 

4.4 

4.3 

0.0 

4.3 

4.2 

0.0 

Academic efficacy 

2.9 

3.0 

0.0 

3.0 

2.8 

0.1 * 

Motivation 

Motivation to learn (7 = intrinsic, -7 = extrinsic) 

-0.8 

-0.2 

I 

O 

ON 

* 

-0.6 

-0.7 

0.1 

Mastery goal orientation 

4.6 

4.5 

0.1 

4.4 

4.4 

0.0 

Performance-avoidance goal orientation 

3.0 

2.7 

0.3 * 

3.2 

3.1 

0.1 

Sample size (total = 511) 

106 

93 


156 

156 



SOURCE: MDRC calculations from the Family Rewards 30-Month Embedded Study Survey. 


NOTES: Statistical significance levels indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Differences across subgroup impacts were tested for statistical significance. Statistical significance levels are indicated as follows: f f f = 1 
percent; f f = 5 percent; f = 10 percent. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre-random assignment characteristics of family or 
sample members. 

Rounding may cause slight discrepancies in calculating sums and differences. 

Sample sizes may vary because of missing values. 

A two-tailed t-test was applied to the differences between outcomes for the program and control groups. 



The Opportunity NYC Demonstration: Family Rewards 
Table A.4 


Impacts on Teenagers' Mental Health and Problem Behaviors, by Proficiency Level on 8th-Grade Math Test 



Scored at or above proficiency level 
on annual math test in 8th grade 

Scored below proficiency level 
on annual math test in 8th grade 


Program 

Control 

Difference 

Program 

Control 

Difference 

Outcome 

Group 

Group 

(Impact) 

Group 

Group 

(Impact) 

Mental health outcomes tO to 13 scale) 

Depression (0 to 13 scale) 

1.4 

i.i 

0.3 

1.8 

1.8 

0.0 

Anxiety (0 to 13 scale) 

3.5 

4.1 

-0.6 

3.7 

3.9 

-0.2 

Problem behaviors 

Delinquent behaviors (%) 

43.4 

50.6 

-7.2 

47.5 

54.4 

-6.9 

Aggressive behaviors (%) 

14.0 

21.7 

-7.6 

14.1 

25.6 

-11.5 ** 

Any substance use in past month (%) 

10.8 

29.6 

-18.8 *** 

21.2 

35.2 

-14 o *** 

Ever smoked cigarettes (%) 

1.7 

1.3 

0.3 

4.8 

9.9 

-5.1 * 

Ever drank alcohol (%) 

8.1 

28.4 

-20.3 *** 

15.9 

25.8 

-10.0 ** 

Ever used marijuana (%) 

3.8 

8.6 

-4.8 

8.0 

17.1 

-9.1 ** 

Ever used hard drugs (%) 

0.0 

0.0 

0.0 

0.6 

0.7 

-0.1 

Number of friends using substances in past month 

(1 = none, 2 = a few, 3 = some, 4 = most) 

1.8 

2.0 

-0.2 * 

1.8 

2.0 

_0 2 *** 

Number of friends drinking alcohol 

2.2 

2.4 

-0.2 

2.2 

2.4 

-0.3 ** 

Number of friends using marijuana 

2.1 

2.4 

-0.3 ** 

2.1 

2.3 

-0.3 ** 

Number of friends using hard drugs 

1.2 

1.2 

-0.1 

1.2 

1.3 

-0.1 

Sample size (total = 511) 

106 

93 


156 

156 



SOURCE: MDRC calculations from the Family Rewards 30-Month Embedded Study Survey. 


NOTES: Statistical significance levels indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Differences across subgroup impacts were tested for statistical significance. Statistical significance levels are indicated as follows: f f f = 1 percent; f f = 5 
percent; f = 10 percent. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre -random assignment characteristics of family or sample members. 
Rounding may cause slight discrepancies in calculating sums and differences. 

Sample sizes may vary because of missing values. 

A two-tailed t-test was applied to the differences between outcomes for the program and control groups. 




Appendix B 

Supplementary Tables for Chapters 2 and 3: Gender 

Subgroup Findings 




The Opportunity NYC Demonstration: Family Rewards 

Table B.l 


Impact on Hours Spent in Weekday Activities, for Girls and Boys 




Girls 



Boys 


Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

Program 

Group 

Control 

Group 

Difference 
(Impact) " 

Kind of activity 

Productive 

2.3 

2.4 

-0.1 

2.4 

2.4 

-0.1 

Productive educational 

1.8 

1.7 

0.1 

1.4 

1.4 

0.0 

Productive work 

0.1 

0.3 

-0.1 

0.3 

0.2 

0.1 

Productive religious 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

Maintenance 

2.5 

2.4 

0.0 

2.1 

1.8 

0.3 

Leisure 

3.3 

3.0 

0.2 

3.7 

3.9 

-0.2 

Accompanying individual during activity 

Alone 

1.2 

1.3 

0.0 

1.4 

1.7 

-0.3 

Family 

4.9 

4.2 

0.7 

4.3 

3.5 

0.8 * 

Peers 

2.1 

2.2 

-0.1 

2.5 

3.1 

-0.6 

Coworkers 

0.1 

0.2 

-0.1 

0.2 

0.2 

0.0 

Teacher 

0.8 

1.0 

-0.2 

0.9 

0.9 

-0.1 

Sample size (total = 510) 

141 

134 


121 

114 



SOURCE: MDRC calculations from the Family Rewards 30-Month Embedded Study Survey. 

NOTES: Statistical significance levels indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Differences across subgroup impacts were tested for statistical significance. Statistical significance levels are indicated as follows: f f f 
= 1 percent; f f = 5 percent; f = 10 percent. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre-random assignment characteristics of family or 
sample members. 

Rounding may cause slight discrepancies in calculating sums and differences. 

Sample sizes may vary because of missing values. 

One teenager did not report gender. This teenager is not included in these analyses. 

A two-tailed t-test was applied to the differences between outcomes for the program and control groups. 



The Opportunity NYC Demonstration: Family Rewards 
Table B.2 

Impacts on Parents' and Teenagers' Spending and Saving, for Girls and Boys 


Girls Boys 


Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

Program 

Group 

Control 

Group 

Difference 

(Impact) 


Sig. 

Parent report 

Total spending in 1 month 3 ($) 

Total spending 

712 

614 

9g ** 

672 

641 

32 

## 

#### 

Productive spending 

165 

111 

54 *** 

144 

141 

3 

## 

ft 

Maintenance spending 

121 

87 

34 ** 

95 

77 

18 

## 

#### 

Leisure spending 

429 

417 

12 #REF! 

437 

423 

14 

## 

#### 

Saving and giving 

Save for child's future education (%) 

60.2 

38.1 

22.2 *** 

45.7 

46.3 

- 0.6 

## 

ft 

How often do you set aside money? 

2.3 

1.8 

0.4 ** 

1.9 

1.9 

0.0 

## 

#### 

( 1 = never, 5 = every week) 

How much money do you set aside? (S) 

176 

95 

81 #REF! 

58 

53 

5 

## 

#### 

Anyone help pay expenses for child? (%) 

30.8 

24.5 

6.3 #REF! 

27.5 

18.2 

9.3 

* 

#### 

How much help was provided this year? ($) 

255 

128 

127 #REF! 

101 

54 

47 

## 

#### 

Teenager report 

Total spending in 1 month * 5 ($) 

Total spending 

195 

295 

-100 ** 

271 

307 

-36 

## 

#### 

Productive spending 

29 

38 

-9 #REF! 

36 

40 

-3 

## 

#### 

Maintenance spending 

145 

205 

-60 ** 

182 

222 

-40 

## 

#### 

Leisure spending 

21 

52 

_3 1 *** 

55 

45 

9 

## 

ft 

Saving and giving 

Save money (%) 

72.3 

67.7 

4.6 #REF! 

81.0 

71.5 

9.5 

## 

#### 

How much did you save? (S) 

61 

63 

-2 #REF! 

123 

84 

40 

## 

#### 

Give money as gift, loan, donation? (%) 

44.2 

45.3 

-1.0 #REF! 

39.6 

38.7 

0.9 

## 

#### 

How much did you give? (S) 

13 

16 

-3 #REF! 

20 

17 

3 

## 

#### 

Sample size (total = 510) 

141 

134 


121 

114 





(continued) 



Table B.2 (continued) 

SOURCE: MDRC calculations from the Family Rewards 30-Month Embedded Study Survey. 

NOTES: Statistical significance levels indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Differences across subgroup impacts were tested for statistical significance. Statistical significance levels are indicated as follows: f f f = 1 percent; 
ff = 5 percent; f = 10 percent. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre-random assignment characteristics of family or sample members. 
Rounding may cause slight discrepancies in calculating sums and differences. 

Sample sizes may vary because of missing values. 

A two-tailed t-test was applied to the differences between outcomes for the program and control groups. 

One teenager did not report gender. This teenager is not included in these analyses. 

“Spending on different items was reported for varying time periods ranging from 1 to 6 months. All numbers reported in this table are scaled to a 1 - 
month interval. 

b Spending on different items was specifically asked about "in the past 4 weeks." 



The Opportunity NYC Demonstration: Family Rewards 

Table B.3 


Impacts on Parent-Teenager Interactions, for Girls and Boys 




Girls 



Boys 


Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

Sig. 

Monitorine/disclosure 

Parental monitoring (5 = high monitoring) 

4.4 

4.4 

0.0 

4.2 

4.3 

0.0 


Teenager disclosure (5 = high disclosure) 

4.0 

3.9 

0.0 

3.7 

3.7 

0.0 


Intensity level of family conflict (5 = ansrv, 1 = calm) 

Parent report 

Discussion of all issues discussed 

1.9 

2.0 

0.0 

2.2 

2.2 

0.0 


Discussion of money-related issues 

1.8 

1.8 

0.1 

1.9 

1.9 

0.0 


Discussion of school-related issues 

2.1 

2.2 

-0.1 

2.7 

2.4 

0.2 


Teenager report 

Discussion of all issues discussed 

2.0 

2.1 

-0.1 

1.9 

1.9 

0.1 


Discussion of money-related issues 

1.8 

2.0 

-0.2 

1.8 

1.6 

0.2 

tf 

Discussion of school-related issues 

2.1 

2.3 

I 

© 

* 

2.1 

2.1 

0.0 


Sample size (total = 510) 

141 

134 


121 

114 




SOURCE: MDRC calculations from the Family Rewards 30-Month Embedded Study Survey. 

NOTES: Statistical significance levels indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Differences across subgroup impacts were tested for statistical significance. Statistical significance levels are indicated as follows: f f f = 
1 percent; f f = 5 percent; f = 10 percent. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre -random assignment characteristics of family or 
sample members. 

Rounding may cause slight discrepancies in calculating sums and differences. 

Sample sizes may vary because of missing values. 

One teenager did not report gender. This teenager is not included in these analyses. 

A two-tailed t-test was applied to the differences between outcomes for the program and control groups. 



The Opportunity NYC Demonstration: Family Rewards 

Table B.4 


Impacts on Teenagers' Approaches to Schooling, for Girls and Boys 




Girls 



Boys 



Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

Sig. 

Academic expectations (%) 

Elope to complete some college 

92.0 

96.4 

-4.4 

88.2 

85.3 

2.9 


Think will complete some college 

95.6 

95.4 

0.2 

85.1 

86.9 

-1.8 


Elope to finish college 

89.0 

91.2 

-2.2 

80.6 

78.3 

2.2 


Think will finish college 

94.1 

88.6 

5.5 * 

76.8 

78.4 

-1.7 


School processes 

School behavioral engagement 

4.5 

4.4 

0.1 

4.2 

4.2 

0.0 


Academic efficacy 

3.0 

2.9 

0.0 

2.9 

2.9 

0.0 


Motivation 

Motivation to leam (7 = intrinsic, -7 = extrinsic) 

3.0 

2.9 

0.1 

3.2 

3.1 

0.0 


Mastery goal orientation 

4.6 

4.5 

0.1 

4.3 

4.4 

0.0 


Performance-avoidance goal orientation 

3.1 

2.8 

0.3 * 

3.2 

3.1 

0.0 


Sample size (total = 510) 

141 

134 


121 

114 




SOURCE: MDRC calculations from the Family Rewards 30-Month Embedded Study Survey. 

NOTES: Statistical significance levels indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Differences across subgroup impacts were tested for statistical significance. Statistical significance levels are indicated as follows: 
ttt = 1 percent; ft = 5 percent; f = 10 percent. 

Estimates were regression-adjusted using ordinaiy least squares, controlling for pre-random assignment characteristics of family or sample 
members. 

Rounding may cause slight discrepancies in calculating sums and differences. 

Sample sizes may vary because of missing values. 

One teenager did not report gender. This teenager is not included in these analyses. 

A two-tailed t-test was applied to the differences between outcomes for the program and control groups. 



The Opportunity NYC Demonstration: Family Rewards 
Table B.5 


Impacts on Teenagers' Mental Health and Problem Behaviors, for Girls and Boys 




Girls 


Boys 



Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

Sig. 

Mental health outcomes 

Depression (0 to 13 scale) 

2.0 

1.8 

0.1 

1.3 

1.1 

0.2 


Anxiety (0 to 13 scale) 

4.1 

4.5 

-0.4 

3.1 

3.2 

-0.2 


Problem behaviors 

Delinquent behaviors (%) 

40.6 

46.1 

-5.5 

53.4 

59.1 

-5.7 


Aggressive behaviors (%) 

11.8 

18.9 

-7.2 

16.9 

30.4 

-13.5 ** 


Any substance use in past month (%) 

13.6 

31.2 

-17.7 *** 

22.1 

33.5 

-11.4 * 


Ever smoked cigarettes (%) 

2.6 

7.0 

-4.4 * 

5.3 

5.8 

-0.5 


Ever drank alcohol (%) 

11.0 

26.5 

-15.5 *** 

14.4 

26.8 

-12.3 ** 


Ever used marijuana (%) 

2.9 

14.1 

-11.2 *** 

11.2 

11.8 

-0.6 

t 

Ever used hard drugs (%) 

-0.1 

0.8 

-0.9 

0.8 

0.0 

0.8 


Number of friends using substances in past month 

( 1 =enone, 2 = a few, 3 = some, 4 = most) 

1.8 

2.0 

-0.2 ** 

1.8 

1.9 

-0.1 


Number of friends drinking alcohol 

2.1 

2.4 

-0.3 ** 

2.2 

2.4 

-0.1 


Number of friends using marijuana 

2.1 

2.4 

-0.4 *** 

2.1 

2.2 

-0.1 


Number of friends using hard drugs 

1.2 

1.3 

-0.1 

1.2 

1.2 

0.0 


Sample size (total = 510) 

141 

134 


121 

114 




(continued) 


Table B.5 (continued) 

SOURCE: MDRC calculations from the Family Rewards 30-Month Embedded Study Survey. 

NOTES: Statistical significance levels indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Differences across subgroup impacts were tested for statistical significance. Statistical significance levels are indicated as follows: ttt = 1 
percent; = 5 percent; f = 10 percent. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre-random assignment characteristics of family or sample 
members. 

Rounding may cause slight discrepancies in calculating sums and differences. 

Sample sizes may vary because of missing values. 

One teenager did not report gender. This teenager is not included in these analyses. 

A two-tailed t-test was applied to the differences between outcomes for the program and control groups. 






Appendix C 

Additional Analyses on Measuring Teenagers’ Time Use 




This appendix includes two additional sets of analyses. The first includes a more detailed 
explanation of the cluster analysis used to assess teenagers’ time use, and the second includes 
additional analyses on teenagers’ weekend time use. 


Cluster Analysis 

Teenagers were asked about the activities they engaged in beginning at 2 p.m. on a typ- 
ical weekday until they went to bed. Questions began with: “What were you doing at 2:00 p.m. 
[YESTERDAY/THURSDAY]?” Activities were coded into one of 23 categories by the 
interviewer. The interviewer then asked how long the reported activity took, where they were 
during this activity, and whom they were with (alone, with parents, with friends, with boy- 
friend/girlfriend, with coworkers, with family members other than parents, with teachers, or 
with someone else). The interviewer then prompted for the next activity by asking: “What did 
you do next on [YESTERDAY/THURSDAY]?” and the same series of follow-up questions 
were asked for this activity. This line of questioning continued until the teenagers reported 
going to sleep for the night. All activities were later categorized in the following way: 

• Academic activities (reading, schoolwork/studying; in class/school; academ- 
ic lessons or activities) 

• Social (hanging out with friends/family; talking or texting on the phone; hav- 
ing sex) 

• Religious activities (Praying/worshiping/meditating/attending church) 

• Family chores/work (doing housework/chores; taking care of chil- 
dren/siblings) 

• Work (working for pay) 

• Organized activities (clubs for nonacademic activities/lessons/sport prac- 
tice) 

• TV/computer (watching TV or videos; computer/Internet/e-mail) 

• Maintenance (self-care or grooming; eating or drinking; exercise; preparing 
meals or snacks; commuting, traveling to and from school or work; nap- 
ping/resting; relaxing; went to bed; shopping; commuting to other places) 
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The total time spent in each activity was summed within categories. All variables were 
standardized to calculate the total time in each activity as a fraction of total time reported on. All 
variables ranged from 0 to 1 . 

The cluster analysis was conducted using pooled data across program and control 
groups; it gives infomiation about postrandomization patterns of time use. The following steps 
were taken as recommended by Milligan, 1 so the cluster analysis relied on both hierarchical and 
iterative methods. There is no assumption of normally distributed data or of uncorrelated 
variables for this clustering method, although all variables do need to have the same range, 
which was accomplished by scaling each as a proportion of total time awake. 

The first step of hierarchical clustering uses Ward’s minimum- variance clustering 
method, which is an agglomerative hierarchical clustering method 2 designed to generate clusters 
in such a way that the variance within the clusters is minimal. In this method, each individual 
case begins in its own cluster. The proximity between every possible pair of individuals is 
calculated using the squared Euclidean distance, in an N-dimensional space, where N is equal to 
the number of variables on which subjects are clustered; in this case N = 8. The Euclidean 
distance was calculated by first finding the difference between the scores of every two subjects 
on each variable included in the cluster analysis, squaring each of the differences, and then 
summing them all. 3 Ward’s method then searches for the two individuals with the smallest 
squared Euclidean distance, and those cases are then merged to form a new cluster that replaces 
the two individual ones. It continues to merge cases in this way until the number of requested 
clusters is reached. 4 

Results from the clusters identified in the hierarchical analysis are then used to inform 
the starting values for clusters in the iterative solution, 5 using a k-means algorithm for determin- 
ing cluster membership, which has been proven to produce superior recovery of cluster struc- 
tures compared with other methods 6 by allowing subjects to change clusters over iterations. The 
iterative cluster analysis aims to identify the most unique cluster solution by minimizing the 
distance within clusters and maximizing the distance between clusters. 7 Once the number of 
clusters was selected based on the hierarchical solution, the iterative analysis was run. 


'Milligan (1996). 

2 ln SAS, the PROC CLUSTER procedure. 
3 Blashfield (1976). 

4 Blashfield (1976); Borgen and Barnett (1987). 
5 In SAS, the FASTCLUS procedure. 

Milligan (1996). 


7 Blashfield (1976). 
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Identification of Clusters 

Figure C.l presents the mean scores of all activity dimensions for each cluster at each stage. At 
the two-cluster solution, teenagers were divided primarily by the amount of time they spent in 
social activities, with Cluster 1-2 spending very little time (6 percent) in social activities and 
Cluster 2-2 spending over half their time in such activities (53 percent). The nonsocial cluster 
(1-2) also spent more time in academic activities. At the three-cluster solution, Cluster 1-2 is 
divided into teenagers who spent a majority of their time engaged in watching TV or on the 
computer (Cluster 2-3), compared with low levels of TV and computer time (Cluster 1-3). This 
latter cluster also spent more time in academic and maintenance activities. At the four-cluster 
solution, Cluster 1-3 is divided into two groups (1-4 and 2-4) of teenagers based on their time 
spent in academic activities; those who spent less time in academic activities (1-4) had higher 
levels of organized work and maintenance activities. Each cluster explained an additional 18 
percent, 1 1 percent, and 9 percent, respectively, of the variance in patterns of time use. 

The five-cluster solution divided Cluster 1 -4 into those who engaged in higher levels of 
maintenance activities (Cluster 1-5) and those who were engaged in high levels of organized 
activities (Cluster 2-5). The fifth cluster explained an additional 6 percent of the variation. 
Because of the very small sample size in Cluster 2-5 and because the differentiation on mainte- 
nance was not a central interest with regard to the Family Rewards program, it was decided that 
the four-cluster solution would be the focus of the impact analyses. 

The iterative analysis was conducted with the results from the hierarchical analysis as 
the cluster seeds. This resulted in four clusters that could be differentiated by their participation 
in activities 8 (see Table C.l). The first cluster predominantly engaged in maintenance activities 
(about 41 percent of their time) and engaged in the highest levels of work and family chores and 
organized activities compared with any other cluster. The second cluster engaged predominant- 
ly in academic activities (about 51 percent of their time). The third cluster engaged predomi- 
nantly in social activities (approximately 53 percent of their time), and the fourth cluster 
engaged predominantly in watching TV and being on the computer (nearly 57 percent of their 
time). The levels of the other activities were somewhat similar across the groups. See Table C.l 
for a more detailed breakdown of each cluster. The four clusters of teenagers were labeled (1) 
maintenance and work-oriented, (2) academically oriented, (3) TV/computer-oriented, and (4) 
socially oriented. 


8 Note that that clusters are presented in a slightly different order in the tables than in Figure C.l. 
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The Opportunity NYC Demonstration: Family Rewards 

Figure C.l 

Results of Hierarchical Cluster Analysis: Proportion of Time Spent in Activities (N = 504) 


Cluster 1/4, N = 197 

Academic 

Work 

Religious activity 
Social 

Family chores 
Organized activities 
TV/computer 
Maintenance 


Cluster 1/2, N = 418 


Academic 

23.1% 

Work 

3.4% 

Religious activity 

0.3% 

Social 

6.0% 

Family chores 

3.6% 

Organized activities 

7.1% 

TV/computer 

26.6 

Maintenance 

27.6% 


Cluster 1/3, N = 304 



Cluster 2/3, N = 114 


Academic 

26.6% 


Academic 

13.7% 

Work 

4.5% 


Work 

0.6% 

Religious activity 

0.5% 


Religious activity 

0.0% 

Social 

6.4% 


Social 

5.0% 

Family chores 

4.6% 


Family chores 

1.0% 

Organized activities 

9.7% 


Organized activities 

0.2% 

TV/computer 

14.6% 


TV/computer 

58.6% 

Maintenance 

30.8% 


Maintenance 

n 

18.9% 


Cluster 2/4, N = 107 

Academic 

Work 

Religious activity 
Social 

Family chores 
Organized activities 
TV/computer 
Maintenance 


Cluster 3/4, N = 114 

Academic 

Work 

Religious activity 
Social 

Family chores 
Organized activities 
TV/computer 
Maintenance 


Cluster 2/2, N = 86 


Academic 

5.5% 

Work 

0.02% 

Religious activity 

0.3% 

Social 

53.9% 

Family chores 

1.3% 

Organized activities 

3.4% 

TV/computer 

18.5% 

Maintenance 

16.0% 


Cluster 3/3, N = 86 


Academic 

5.5% 

Work 

0.02% 

Religious activity 

0.3% 

Social 

53.9% 

Family chores 

1.3% 

Organized activities 

3.4% 

TV/computer 

18.5% 

Maintenance 

16.0% 


Cluster 4/4, N = 86 


Academic 

5.5% 

Work 

0.02% 

Religious activity 

0.3% 

Social 

53.9% 

Family chores 

1.3% 

Organized activities 

3.4% 

TV/computer 

18.5% 

Maintenance 

16.0% 


Cluster 1/5, N = 141 

Academic 

Work 

Religious activity 
Social 

Family chores 
Organized activities 
TV/computer 
Maintenance 


Cluster 2/5, N = 56 

Academic 

Work 

Religious activity 
Social 

Family chores 
Organized activities 
TV/computer 
Maintenance 


Cluster 3/5, N = 107 

Academic 

Work 

Religious activity 
Social 

Family chores 
Organized activities 
TV/computer 
Maintenance 


Cluster 4/5, N = 114 

Academic 

Work 

Religious activity 
Social 

Family chores 
Organized activities 
TV/computer 
Maintenance 


Cluster 5/5, N = 86 

Academic 

Work 

Religious activity 
Social 

Family chores 
Organized activities 
TV/computer 
Maintenance 




The Opportunity NYC Demonstration: Family Rewards 
Table C.l 

Proportion of Time Spent in Activities by Time Use Group 


Outcome 

Maintenance/ 
Work Oriented 

Academically 

Oriented 

TV/Computer 

Oriented 

Socially 

Oriented 

Proportion of time 

Academic 

0.11 

(0.11) 

0.51 

(0.17) 

0.15 

(0.13) 

0.07 

(0.09) 

Work 

0.07 

(0.19) 

0.02 

(0.08) 

0.01 

(0.04) 

0.01 

(0.08) 

Religious activity 

0.01 

(0.06) 

0.00 

(0) 

0.00 

(0) 

0.00 

(0.03) 

Social 

0.05 

(0.09) 

0.06 

(0.09) 

0.05 

(0.09) 

0.53 

(0.16) 

Family chores 

0.06 

(0.17) 

0.02 

(0.08) 

0.02 

(0.07) 

0.01 

(0.05) 

Organized activities 

0.14 

(0.23) 

0.03 

(0.1) 

0.03 

(0.08) 

0.03 

(0.08) 

TV/computer 

0.12 

(0.12) 

0.14 

(0.13) 

0.57 

(0.16) 

0.15 

(0.13) 

Maintenance 

0.41 

(0.25) 

0.21 

(0.11) 

0.17 

(0.11) 

0.19 

(0.12) 

Sample size (total = 504) 

157 


115 


137 


95 



SOURCE: MDRC calculations from the Family Rewards 30-Month Embedded Study Survey. 

NOTES: The table presents time use group means. Standard deviations for each activity are presented in 
parentheses. 

Activities with highest levels are indicated in bold. 

Rounding may cause slight discrepancies in calculating sums and differences. 

Sample sizes may vary because of missing values. 


Sensitivity and Validity Analyses 

Three sets of sensitivity and validity analyses were conducted. First, the sensitivity of the 
clusters to the addition of particular variables was tested. Second, experimental-control differ- 
ences in cluster identification and in distance from cluster centers were tested to identify 
whether the clusters better represented the program group or the control group. Finally, the 
validity of the clusters was tested by examining the association with clusters and outcomes for 
adolescents. Each of these analyses is discussed in turn below. 

Sensitivity to the Inclusion of Particular Variables 

While most of the items that comprised the eight categories were face valid and func- 
tionally equivalent within categories, there were two exceptions. First, time spent on the 
computer was categorized together with TV time. Given that time spent on the computer may 
be productive (for example, doing homework or research on the Internet for a school assign- 
ment), nonproductive (playing video games, watching YouTube), or social (e-mailing/chatting 
with friends), the validity of the cluster structure was tested by separating these two items. 
Second, the maintenance category was comprised predominantly of self-care and grooming 
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activities, eating, and napping, but also included time spent in commuting. The validity of the 
cluster structure was tested by separating these two sets of items. 

First, the cluster analysis was run separating TV from computer activities. The four- 
cluster solution produced fairly similar clusters to the original model. For the TV/computer- 
oriented cluster, watching TV made up 56 percent of their total time, while time on the comput- 
er was only about 6 percent of the total time, suggesting that this cluster is primarily made up of 
teenagers watching television. 

Next, the cluster analysis was run separating time spent in commuting from the rest of 
the maintenance items. This analysis produced a four-cluster solution similar to the original 
model. In the “maintenance/work” cluster, commuting accounted for only about 5 percent of 
that cluster’s total time, while napping, resting, eating, and self-care together accounted for 70 
percent of the total time. Thus, most of the maintenance time was time spent in other mainte- 
nance activities (napping, resting, eating, self-care), rather than commuting. 

Cluster Analysis by Experimental Condition 

Because time use was measured after random assigmnent and after the beginning of 
treatment implementation, the cluster analysis in both the experimental and control groups was 
run separately to test whether the results of the analysis were similar. If the cluster structure was 
an artifact of the treatment, one would expect to see different results in the two groups. In both 
conditions, the final four groups in the iterative solution were similar to those of the main 
analysis, with one that was predominantly engaged in maintenance activities, a second that was 
predominantly engaged in academic activities, a third that was predominantly engaged in TV 
and computer time, and a fourth that was predominantly engaged in social activities. In both 
conditions within all four clusters, adolescents spent more time in the predominant activity by a 
factor of two or three from the next highest activity (see Table C.2). 

Impacts on Distance from Cluster Centroid 

By looking at differences in the average distance from the cluster centroid (that is, clus- 
ter center), it is possible to test if the clusters better represent one experimental group over the 
other. If there is no difference between how far, on average, teenagers are from their cluster 
center within clusters, one can be confident that each of the clusters represent both groups 
equally well. These analyses include the same baseline covariates that were included in every 
model. There were no significant differences between program and control teenagers on 
average distance from cluster centroid. 
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The Opportunity NYC Demonstration: Family Rewards 
Table C.2 


Proportion of Time Spent in Activities by Time Use Group, by Experimental Condition 


Outcome 


Program Group 




Control Group 



Maintenance/ 
Work Oriented 

Academically TV/Computer 
Oriented Oriented 

Socially 

Oriented 

Maintenance/ 
Work Oriented 

Academically TV/Computer 
Oriented Oriented 

Socially 

Oriented 

ProDortion of time (%) 









Academic 

9.5 

51.2 

12.8 

5.5 

11.6 

50.0 

16.7 

7.5 

Work 

7.4 

1.3 

0.7 

1.5 

6.2 

2.0 

0.5 

1.0 

Religious activity 

0.6 

0.0 

0.0 

0.6 

1.3 

0.0 

0.0 

0.0 

Social 

4.8 

7.3 

5.3 

54.9 

5.0 

4.8 

4.2 

50.9 

Family chores 

6.7 

2.8 

2.2 

2.0 

5.6 

1.4 

1.5 

1.1 

Organized activities 

15.4 

2.1 

2.1 

2.3 

13.4 

5.1 

3.3 

2.8 

TV/computer 

10.7 

12.9 

57.5 

13.1 

13.4 

15.6 

55.8 

16.8 

Maintenance 

40.7 

20.9 

18.0 

17.8 

40.8 

20.4 

15.9 

19.9 


SOURCE: MDRC calculations from the Family Rewards 30-Month Embedded Study Survey. 

NOTES: Activities with highest levels are indicated in bold. 

Rounding may cause slight discrepancies in calculating sums and differences. 



Cluster Validity Check 

To test the convergent validity of the clusters (that is, that cluster membership related to 
outcomes that it should theoretically be related to), cluster membership was regressed on a 
variety of academic and behavioral outcomes. Teenagers who spent more time in academics 
(that is, the academically oriented cluster) were predicted to have better academic outcomes, 
while teenagers who spent more time in social activities (that is, the socially oriented cluster) 
were predicted to be more likely to engage in problematic behaviors. In addition, teenagers who 
spent more time in leisure activities were predicted to spend less time in structured extracurricu- 
lar activities. Table C.3 confirms these predictions. 


Program Impacts Using a Multinomial Logistic Regression 

A multinomial logistic regression was conducted to determine whether program status predicted 
cluster membership, using the “academically oriented” cluster as the reference group. All 
covariates were included. While in the overall model, program status was not significant (b = 
5.31, p = 0.15), planned pair-wise comparisons of each of the three clusters with the academi- 
cally oriented cluster revealed that program status was a significant predictor for the socially 
oriented compared with the academically oriented cluster (b = -0.68, p < .05). This indicates that 
Family Rewards increased the proportion of teenagers in an academically oriented group and 
reduced the proportion in a socially oriented group. Estimates for the other two clusters were in 
the same direction but are not statistically significant (for the maintenance/work-oriented 
cluster, b = -0.43, p = 0.104; for the TV/computer cluster, b = -0.30, p = 0.27). 

Subgroup Impacts by Proficiency and Gender 

A multinomial logistic regression was conducted on two sets of subgroups: those that 
were academically prepared at the start of the study and those that were not (that is, proficient 
versus nonproficient at baseline), and males and females. The analysis of proficiency subgroups 
revealed that program impacts found in the overall sample were concentrated among those that 
were proficient at baseline (see Chapter 2). The coefficient for program status in the model was 
significant (b = 9.55, p < 0.05), and planned pair-wise comparisons of each of the three clusters 
revealed that there were program group differences in cluster membership. Specifically, 
program status was a significant predictor for the socially oriented cluster compared with the 
academically oriented cluster (b = 1.44, p < 0.05), and program status was a significant predic- 
tor for the maintenance/work-oriented cluster compared with the academically oriented cluster 
(b = 0.76, p < 0.10). This indicates that Family Rewards increased the proportion of teenagers 
in the academically oriented group and reduced the proportion in the socially oriented and 
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Table C.3 


Predictors of Outcomes by Cluster Membership 


Outcome 

Academic vs. 
Maintenance/ 
Work 

Academic vs. 
TV/Computer 

Academic vs. 
Social 


Academic outcomes 

Attendance, Year 1 

-0.07 

-0.06 

-0.12 

** 

Attendance, Y ear 2 

-0.11 * 

-0.10 * 

-0.17 

*** 

Took 1 1 credits, Year 1 

0.01 

-0.04 

-0.04 


Took 1 1 credits, Year 2 

-0.02 

-0.04 

-0.03 


Passed 1 1 credits, Year 1 

-0.07 

-0.09 * 

-0.14 

*** 

Passed 22 credits, Year 1 to 2 

-0.13 ** 

-0.12 ** 

-0.22 

*** 

Passed at least 1 Regents exam, Y ear 1 

-0.04 

-0.09 * 

-0.11 

** 

Passed at least 1 Regents exam, Y ear 2 

-0.02 

-0.06 

-0.08 

* 

Number of Regents exams passed 

-0.03 

-0.08 * 

-0.11 

** 

Problem behaviors 

Delinquent behaviors (ever) 

0.04 

0.04 

0.09 


Aggressive behaviors (ever) 

-0.08 

-0.11 * 

0.04 


Any individual substance use 

-0.03 

-0.01 

0.16 

*** 

Any peer substance use 

0.12 ** 

0.10 * 

0.13 

** 

Mental health 

Depression/anxiety 

-0.03 

0.06 

0.06 


Motivational and behavioral outcomes 

School engagement 

-0.11 ** 

-0.08 

-0.15 

*** 

Academic efficacy 

0.10 

-0.02 

0.04 


Frequency of participation in organized activities 8 

-0 15 *** 

-0 28 *** 

-0.20 

*** 

Number of organized activities 

-0.10 * 

-0.25 *** 

-0.12 

** 

Attended program for schoolwork 8 

_0 2 i *** 

-0 23 *** 

-0.17 

*** 

Sample size (total = 504) 






SOURCE: MDRC calculations from the Family Rewards 30-Month Embedded Study Survey. 


NOTES: Statistical significance levels indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

All estimates are standardized regression coefficients. 

Activities with highest levels are indicated in bold. 

Academics is the reference group. 

Covariates and program condition are controlled for in models. 

“Denotes the average frequency of participation: 1 = never, 2 = less than once a month, 3 = every month, 4 = 
every week, 5 = every day. 
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maintenance/work-oriented groups for those teenagers who were proficient at baseline and did 
not change time use patterns for teenagers who were not proficient at baseline. 

The analysis of gender subgroups revealed that program impacts were concentrated in 
the girls only. While the overall regression was not significant, the coefficient for program 
status in the model was significant (b = 6.64, p < 0.10), and planned pair-wise comparisons of 
each of the three clusters revealed that there were program group differences in cluster member- 
ship. Specifically, program status was a significant predictor for the socially oriented compared 
with the academically oriented cluster (b = 0.90, p < 0.05), and program status was a significant 
predictor for the maintenance/work-oriented compared with the academically oriented cluster (b 
= 0.75, p < 0 .05). This indicates that Family Rewards increased the proportion of teenagers in 
the academically oriented group and reduced the proportion in the socially oriented and mainte- 
nance/work-oriented groups for girls and did not change time use patterns for boys. 


Analysis on Weekend Time Use 

To assess weekend time use, a reduced set of questions were asked about participation in and 
length of time spent in each of a set of prespecified activities. Six questions assessed whether 
teenagers participated in six activities on the previous weekend day. These activities included 
working for pay; spending time reading, studying, or receiving academic tutoring; participation 
in sports, clubs, nonacademic lessons, or religious group activities; doing housework or chores 
or taking care of siblings and other children in the household; watching TV, playing video 
games, or playing on the computer; and spending time hanging out with friends and/or family. 
Teenagers were also asked whether there were any other activities they participated in that were 
not mentioned; these responses were hand-coded. Responses were then categorized into three 
overarching categories: productive activities (including the subcategories productive- 
educational and productive-work), maintenance activities, or leisure activities (see below for 
definitions). The total time spent (hours) in each category of activities was summed. 

Activities were categorized similarly to the variable-centered analysis of weekday time 
use into productive (working for pay; reading, studying, or receiving academic tutoring; 
participating in sports, clubs, nonacademic lessons [like music]; or religious group activities), 
maintenance (doing housework or chores or taking care of siblings and other children in the 
household; health-related activities), and leisure (watching TV, playing video games, or playing 
on the computer; hanging out with friends and/or family). Given the goals of the program, for 
the weekday time use categories, productive activities were separated into two subcategories, 
which were productive work (working for pay) and productive educational (reading, study- 
ing, or receiving academic tutoring) activities. 
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Table C.4 


Impacts on Hours Spent in Weekend Activities 


Outcome 

Program 

group 

Control 

group 

Difference 

(Impact) 

P-value 

Productive 

2.0 

2.0 

0.0 

0.915 

Productive educational 

0.2 

0.4 

-0.2 

0.123 

Productive work 

0.2 

0.4 

-0.2 * 

0.073 

Maintenance 

1.6 

1.6 

0.0 

0.952 

Leisure 

6.7 

7.0 

-0.2 

0.641 

Sample size (total = 511) 

262 

249 




SOURCE: MDRC calculations from the Family Rewards 30-Month Embedded Study Survey. 

NOTES: Statistical significance levels indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 
percent. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre -random 
assignment characteristics of family or sample members. 

Rounding may cause slight discrepancies in calculating sums and differences. 

Sample sizes vary because of missing values. 


Table C.4 shows the program impacts on weekend time use. Compared with time use 
reported for a typical weekday afternoon (see Chapter 2), teenagers reported spending slightly 
less time (two hours) on productive activities (although much less time on educationally 
oriented activities, specifically), slightly less time (about one and a half hours) on maintenance 
activities, but substantially more time on leisure activities (approximately seven hours). Howev- 
er, there are no statistically significant differences between teenagers in the program and control 
groups in the amount of time spent in any of these activities. The single exception is that 
teenagers are reporting spending less time working for pay on the weekend in the Family 
Rewards group compared with the control group. (0.2 hours for program group teenagers 
compared with 0.4 hours for control group teenagers). 
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Appendix D 

Description of Outcome Measures Used in the Study 




Appendix D describes the measures of family and child processes that were used to assess the 
impact of Family Rewards on teenagers and families. Results of the impact analysis using these 
measures are presented in Chapters 2 and 3. Three main areas of family and child processes 
were measured — teenagers’ time use, spending and saving, and parent-teenager interactions; 
teenagers’ approaches to schooling; and teenagers’ mental health and behavioral problems. 
When reported, reliability coefficients are the result of the Cronbach’s alpha value for the items 
in each scale. Alpha values great than 0.70 represent high internal reliability of the scale. 


Time Use and Spending and Saving 

Time Use 

Individual Weekday Activities 

Adapted from the American Time-Use Survey, 1 teenagers were asked to report on eve- 
ry activity they participated in between 2 p.m. and bedtime. Teenagers reported on what the 
activity was as well as how much time was spent in each activity. Activities were categorized 
into three overarching categories: productive activities, which are activities that can provide 
positive developmental opportunities to youth (for example, working or in academic lessons or 
activities); maintenance activities, which are often obligatory and include activities done for the 
maintenance and care of the family and oneself (for example napping/resting or doing chores 
and errands); and leisure activities, which include free-time and discretionary activities (for 
example, socializing or watching TV). Within the category of productive activities, subcatego- 
ries were included of productive-educational, productive-work, and productive-religious 
activities. The total time spent (hours) in each of these categories was counted. 

Time Use Groups 

A cluster analysis was conducted to explore whether Family Rewards changed the 
overall way teenagers spent their time. Activities were categorized into academic, work, 
religious, social, family chores, time spent on the computer or watching TV, and maintenance 
activities. This analysis yielded four “clusters” or “groups” of teenagers in the sample. Cluster 
1, the academic group, spent over half of their time engaged in academic activities on average, a 
sizably larger amount than any of the other clusters (compared with 6 to 14 percent). Cluster 2, 
the social group, spent over half of their time engaged in social activities on average, a signifi- 
cantly larger amount than other clusters (compared with 5 to 7 percent). Cluster 3, the 
TV/computer group, spent nearly 60 percent of their time engaged in watching TV or on the 

'For the American Time-Use Survey, see http://www.bls.gov/tus/. 
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computer on average (compared with 13 to 19 percent). Finally cluster 4, the maintenance/work 
group spent more time than any others engaged in maintenance activities (for example, self- 
care, eating, napping, commuting) as well as more time than any others working for pay and 
doing family chores such as housework and taking care of siblings. 2 

Weekend Time Use 

Weekend time use was assessed with six yes/no questions that asked teenagers if they 
participated in six activities on the previous weekend day. Teenagers were also asked if there 
were any other activities they participated in that were not mentioned; these responses were 
hand-coded. Additionally, teenagers were asked how much time they spent in each activity. 
Responses to the six activities and the open-ended question were then categorized into three 
overarching categories: productive activities (including the subcategories productive- 
educational and productive-work), maintenance activities, or leisure activities. The total time 
spent (hours) in each activity within each category was summed. 

Spending 

Parents and teenagers were asked about their spending on a variety of items. Spending 
was coded into three distinct categories, and money spent on each category was summed. 
Productive spending included spending on education-related activities (for example, spending 
on books, school supplies, or other school-related costs) and sports-related activities (for 
example, spending on games or sporting goods). Maintenance spending included spending on 
necessities (for example, spending on food, health care, clothes, or shoes). Finally, leisure 
spending included spending on recreational nonnecessities (for example, entertainment, video 
games, or movies). 

Saving 

Parents’ saving for their child’s future education was asked as a single dichotomous 
question: “Do you set aside money for [FCNAME’S] future education expenses?” Parents 
answered either yes or no. 


2 See Appendix C for more detail on the creation of these groups. 
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Family Interactions 


Family Conflict 

An 11 -question scale 3 was used to measure potential sources of conflict between par- 
ents and teenagers. Parents and teenagers were each asked to report on all 1 1 questions whether 
they (1) discuss the issue and (2) how calm or angry they feel when they discuss the issue (using 
a 5-point scale where 1 = calm and 5 = angry). Parents and teenagers could receive a score of 0 
to 1 1 for the number of issues discussed, and a score from 1 to 5 for the average rating of 
conflict. High scores refer to higher conflict during discussions. 

Parental Monitoring 

A nine-item subscale 4 was used reflecting parents’ tracking and surveillance of their 
teen’s behavior and activities (for example, knowing what their teenager does during free time). 
A four-point response scale was used, with higher scores indicating higher levels of parental 
monitoring. The scale had a reliability coefficient of 0.71, indicating sufficient internal con- 
sistency. 

Teenager Disclosure 

A five-item subscale 5 was used, reflecting the extent to which teenagers tell their par- 
ents about their behavior and activities (for example talking to parents about what you do at 
night/on weekends). A five-point response scale was used, with higher scores indicating higher 
levels of disclosure to parents. The scale had a reliability coefficient of 0.66, indicating suffi- 
cient internal consistency. 


Teenagers’ Approaches to Schooling 

School Engagement 

A four-item scale 6 assessed a teenagers’ engagement in classroom and school activities. 
The four items refer to trying hard to do well, paying attention in class, working hard in school, 
and listening carefully in class. The scale had a reliability coefficient of 0.75, indicating suffi- 
cient internal consistency. 


’Adapted from Robin and Foster (1989). 
4 From Stattin and Kerr (2000). 

5 From Stattin and Kerr (2000). 

6 Adapted from Furrer and Skinner (2003). 
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Academic Efficacy 

A four-item scale 7 assessed teenagers’ perceptions of their academic abilities and ability 
to succeed when faced with challenging academic activities. Teenagers were asked about four 
academic activities (for example, writing a report about a book or passing a final exam at the 
end of the year). A four-point response scale was used, with 1 = “really easy,” 2 = “sort of 
easy,” 3 = “sort of hard,” and 4 = “really hard.” The scale had a reliability coefficient of 0.52, 
indicating relatively low but sufficient internal consistency. 

Motivation 

A 16-item scale assessed the reasons why teens do their schoolwork. Items consisted of 
four subscale scores of different dimensions of motivation (intrinsic, extrinsic, introjected, and 
identified). A score was calculated to create the “Relative Autonomy Index,” based on a 
combination of these subscales ranging from negative seven to positive seven. 8 Larger negative 
scores indicate that teenagers are more extrinsically motivated in doing their schoolwork (for 
example, “I do homework because I will get in trouble if I don’t do it”; “I may get a reward if I 
do it.”) Scores closer to positive seven indicate that teenagers are more intrinsically motivated 
(for example, “I do homework because it’s important for me to do well in school.”). 

Mastery Goal Orientation 

A five-item scale adapted from the Manual for the Patterns of Adaptive Learning 
Scales, 9 measured the extent to which a teenager leams because of a desire to become proficient 
in a topic to the best of his or her ability (for example, learning because it is a goal to leam as 
much as possible or because it is a goal to master a lot of new skills). A five -point response 
scale indicated higher levels being equivalent to higher levels of a mastery goal orientation (1 = 
not at all true, 5 = very true). The scale had a reliability coefficient of 0.77, indicating sufficient 
internal consistency. 

Performance-Avoidant Goal Orientation 

A four-item scale 10 measured the extent to which a teenager behaves in school in order 
to avoid an undesirable outcome (for example, avoiding seeming incompetent to others). A five- 
point response scale indicated higher levels being equivalent to higher levels of a mastery goal 


7 From Seidman et al. (1994). 

8 From Ryan and Connell (1989). For more 
SDT/measures/ SRQacademic.php. 

9 From Midgley et al. (2000). 

10 From Midgley et al. (2000). 


information, see http://www.psych.rochester.edu/ 
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orientation (1 = not at all true, 5 = very true). The scale had a reliability coefficient of 0.72, 
indicating sufficient internal consistency. 


Teenagers’ Mental Health and Problem Behaviors 

Depression 

Thirteen items measuring depressive symptoms were taken from the Behavioral As- 
sessment System for Children. 11 Teenagers answered whether each of the questions were true 
about them since the beginning of the calendar year (for example, “Nothing is fun anymore”; “I 
feel sad.”). Items were summed to create a scale ranging from zero to 13, with higher scores 
indicating higher levels of depression. The scale had a reliability coefficient of 0.86, indicating 
sufficient internal consistency. 

Anxiety 

Thirteen items measuring symptoms of anxiety were taken from the Behavioral As- 
sessment System for Children. 12 Teenagers answered whether each of the questions were true 
about them since the beginning of the calendar year (for example, “I worry about little things”; 
“I get nervous.”). Items were summed to create a scale ranging from zero to 13, with higher 
scores indicating higher levels of anxiety. The scale had a reliability coefficient of 0.80, indicat- 
ing sufficient internal consistency. 

Delinquent Behaviors 

Six items 13 were used to measure the frequency of minor delinquent acts (actions con- 
sidered criminal offenses for adults) committed in “the past couple of weeks” (for example, 
“taking something from someone at school that didn’t belong me,” “breaking or ruining 
something on purpose that belonged to the school”). A four-point response scale was used, with 
higher scores indicating a higher frequency of committing such acts ( 1 = “never,” 4 = “many 
times”). Due to a low frequency in reporting of delinquent behaviors, the scale was averaged 
and then dichotomized to indicate whether a teenager displayed any delinquent activities. 


1 'From Reynolds and Kamphaus (1998). 
12 From Reynolds and Kamphaus (1998). 
13 From Loeber and Dishion (1983). 
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Aggressive Behaviors 

Nine items 14 were used to measure the number of times in the past seven days teenagers 
committed aggressive acts intended to cause harm to others (for example, “teased students to 
make them angry,” “threatened to hurt or hit someone”). A four-point response scale was used, 
with 0 = “0 times,” 1 = “1 time,” 2 = “2 times,” 3 = “3 times,” and 4 = “4 times.” Due to a low 
frequency in reporting of aggressive behaviors, the scale was averaged and then dichotomized 
to indicate whether a teenager displayed any aggressive acts or displayed no delinquent acts. 

Substance Use 

Four items about the frequency with which teenagers smoked cigarettes, drank alcohol, 
used marijuana, and used hard drugs (ecstasy, cocaine, crack, LSD, uppers, or downers) during 
the past month were adapted from the 18-month youth interview in MDRC’s Working toward 
Wellness evaluation 15 and the interview in MDRC’s Supporting Healthy Marriage evaluation. 16 
A four-point response scale was used for all items, with 1 = “never,” 2 = “only a few times,” 3 = 
“1 or 2 times a week,” 4 = “several times a week or more.” Due to a low frequency in reporting 
of substance use, the scale was averaged and then dichotomized to indicate whether a teenager 
ever used substances or never used substances. 

Peer Substance Use 

Three items were used, adapted from the 18-month youth interview in MDRC’s Work- 
ing toward Wellness evaluation 17 and the interview in MDRC’s Supporting Healthy Marriage 
evaluation 18 about how many friends drank alcohol, how many friends tried marijuana, and how 
many friends tried hard drugs in the past month. A four-point scale was used for all items, with 
1 = “none,” 2 = “a few,” 3 = “some,” and 4 = “most.” Items were averaged and a dichotomized 
scale was created to indicate whether a teenager had any peers who used substances. 


14 From Orpinas and Frankowski (2001). 
ls From Kim et al. (2010). 

16 From Miller Gaubert et al. (2010). 
17 From Kim et al. (2010). 

18 From Miller Gaubert et al. (2010). 
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Appendix E 

Analysis of Nonresponse Bias in the 
Child and Family Embedded Study Sample 




This embedded study provides information about families in Family Rewards with a ninth- 
grade child at random assigmnent on topics such as teenagers’ time use, parents’ and teenagers’ 
spending and saving, parent-teenager interactions, teenagers’ approaches to schooling, and 
mental health and problem behaviors. As the survey was administered to a subset of the Family 
Rewards families with a ninth-grade child, it is necessary to assess the reliability of impact 
results for the survey sample along two dimensions. First, the results for the survey sample may 
or may not generalize to (or be representative of) the full sample of families with a ninth-grade 
child, because (1) only a subset of the full Family Rewards sample of families with a ninth- 
grade child was selected to be interviewed; and (2) parents and teenagers who responded to the 
survey may be different from those who were chosen for the survey but did not respond. 
Second, the failure of some families to respond to the survey may compromise the validity of 
the impact estimates, particularly if response rates differ by research group. This appendix 
presents a description of the survey fielding effort and assesses the survey in terms of its 
generalizability to the full sample and its validity for estimating program impacts. Overall, the 
results suggest that the embedded study survey sample (referred to as the child survey sample) 
provides valid estimates of the program’s effects that can be generalized to the full research 
sample. 


Sample Selection and Survey Administration 

From the 1,028 families with a ninth-grade child who had completed the 18-month Family 
Rewards survey, 903 families had infonnation on parent employment at baseline and data on 
teenager baseline math proficiency in the school year before they entered the study. From these, 
903 families, a subsample of 716 teenagers (and their parents) were selected for interviewing 
(the fielded sample). 

Program group members who responded to the 1 8-month survey had better initial em- 
ployment situations than the control group members who responded, so sampling involved 
stratifying the respondent sample by treatment status and parent employment status at baseline. 
The study also found better educational gains for teenagers in the program group who were 
proficient in math in eighth grade. In order to conduct follow-up subgroup analysis from this 
important finding, the sample was also stratified by whether the teenagers met their eighth- 
grade math learning standards. Because there were not enough teenagers who were considered 
proficient in math in eighth grade to stratify the sample with a 50/50 proficient/nonproficient 
split, the subsample was selected so that the percentage of proficient teenagers was balanced 
between the program and control groups. 
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Comparisons Between the Survey Sample and Report Sample 

Analyses comparing the selected sample of parents and teenagers with the full Family Rewards 
sample show that these samples did not differ in important ways, with a few exceptions. Tables 
E.l and E.2 show parents’ and teenagers’ baseline characteristics for the full Family Rewards 
sample (4,749 families), the 18-month survey respondent sample (3,082 families), the families 
fielded for the child survey (716 families), and the families that were eligible for the current 
study but not selected to participate (187 families). The adults not selected for the study were 
more likely to be U.S. citizens by birth (72 percent) and less likely to be legal permanent 
residents (12 percent) than those in the full sample (67 percent and 17 percent, respectively, of 
the full sample). The adults selected for the study were less likely to be Hispanic or Latino and 
more likely to be black than the full sample. Teenagers selected for the study were less likely 
than the teenagers in the full Family Rewards sample to be covered by public health insurance. 
An additional selection criterion established for the purposes of the 18 -month survey was that 
the participating adult in the family either spoke English well or very well or spoke primarily 
English or Spanish in the home. As a result, parents in the 18-month and the embedded study 
survey samples, as well as eligible 1 8-month survey respondents who were not selected for the 
embedded study, were all more likely to primarily speak English than another language in their 
homes than parents in the full research sample. 

A comparison of the last two columns of Table E.l shows that parents in the fielded 
sample differ from parents in the nonfielded sample in a few ways. Because the sample for this 
embedded study was stratified by baseline employment status, adults not selected for the study 
had more employment activity prior to and at baseline than the selected adult sample. It follows 
that they were also more likely to have employer-provided health insurance and less likely to be 
covered by public health insurance than the selected adult sample. Those not selected for the 
study were also 5 percentage points more likely to have an account at a bank or credit union (52 
percent) than the selected sample (47). 

Table E.2 shows similar comparisons of baseline characteristics for the teenagers in 
these families. Teenagers in the fielded sample had characteristics similar to those of teenagers 
who were not selected for the study, although the nonfielded sample did exhibit a slightly 
different ethnic mix than the full Family Rewards sample. 

Fielding of the survey began in March 2010 and ended in June 2010. Members of the 
fielded sample were initially contacted by a letter that introduced the survey and solicited their 
participation. Teenagers were offered $50 and parents were offered $25 for completing the 
survey, which was administered by telephone to those who agreed to participate. Respondents 
were interviewed anywhere between 29 and 35 months after they were randomly assigned. 
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The Opportunity NYC Demonstration: Family Rewards 
Table E.l 


Characteristics of Adult Sample at the Time of Random Assignment, 
by Full, Core Survey Respondent, Child Study, and Not Selected Samples 


Characteristic 

Full 

Sample 

Core Survey 
Respondent 
Sample 

Child Study 
Sample 

Not Selected 
for Child 
Survey 

Female (%) 

94.4 

95.1 

94.3 

95.7 

Age (%) 

18 to 24 years 

0.3 

0.3 

0.0 

0.5 

25 to 34 years 

28.9 

28.4 

17.0 

17.6 

35 to 44 years 

45.1 

45.3 

49.9 

48.1 

45 to 59 years 

22.8 

23.0 

29.7 

32.1 

60 years or more 

2.9 

2.9 

3.4 

1.6 

Average age (years) 

40 

40 

42 

42 

U.S. citizen 2 (%) 

83.1 

85.1 

84.1 

87.7 

By birth 

67.4 

70.7 

70.1 

71.7 

By naturalization 

15.7 

14.4 

14.0 

16.0 

Legal permanent resident (%) 

16.9 

14.9 

15.9 

12.3 

Race/ethnicity (%) 

Hispanic/Latino 

46.7 

43.6 

43.5 

43.0 

White, non-Hispanic/Latino 

0.7 

0.7 

0.4 

0.5 

Black, non-Hispanic/Latino 

51.2 

54.8 

55.1 

56.5 

Other, non-Hispanic/Latino 

1.4 

0.9 

1.0 

0.0 

Education (%) 

GED certificate 

11.1 

11.5 

10.4 

14.9 

High school diploma 

21.3 

21.0 

19.8 

19.6 

Associate's degree/2-year college 

8.9 

8.8 

7.8 

8.8 

4-year college or beyond 

8.7 

9.2 

12.0 

12.2 

None of the above 

50.1 

49.6 

50.0 

44.6 

Marital status (%) 

Single 

62.5 

63.6 

65.4 

60.4 

Cohabiting 

2.3 

2.5 

1.8 

1.1 

Separated, widowed, or divorced 

16.2 

15.5 

15.8 

18.7 

Married or in a legal domestic partnership 

19.1 

18.4 

17.0 

19.8 

Primary language is English (%) 

76.9 

80.6 

81.4 

80.7 

Primary language is Spanish (%) 

21.7 

18.6 

17.9 

18.7 

Has an account at bank or credit union (%) 

47.0 

47.9 

47.0 

51.9 

Employment measures (BIF) 

Currently working (%) 

53.0 

52.6 

50.0 

70.1 

Working full time b (%) 

37.1 

37.3 

37.8 

52.9 

Average weekly earnings, among those currently working ($) 

393 

400 

413 

450 

During past year, average number of months worked (%) 

10 

10 

10 

11 
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Table E.l (continued) 


Characteristic 

Full 

Sample 

Core Survey 
Respondent 
Sample 

Child Study 
Sample 

Not Selected 
for Child 
Survey 

Employment measures (UI) 

Employed in 4th quarter prior to random assignment (%) 

47.5 

47.9 

47.5 

55.6 

Average earnings in 4th quarter prior to random assignment ($) 

2,572 

2,560 

2,647 

3,227 

Employed in 3rd quarter prior to random assignment (%) 

48.8 

49.5 

49.3 

59.4 

Average earnings in 3rd quarter prior to random assignment ($) 

2,772 

2,818 

2,927 

3,935 

Employed in 2nd quarter prior to random assignment (%) 

47.1 

47.2 

47.3 

57.2 

Average earnings in 2nd quarter prior to random assignment ($) 

2,683 

2,692 

2,743 

3,780 

Employed in 1 st quarter prior to random assignment (%) 

47.9 

48.2 

46.8 

60.4 

Average earnings in 1 st quarter prior to random assignment ($) 

2,783 

2,842 

2,861 

3,884 

Public assistance measures 

Received TANF in the year prior to random assignment (%) 

41.3 

40.9 

40.1 

31.0 

Received Medicaid in the 3 quarters prior to 

random assignment (%) 

66.2 

65.6 

65.6 

61.0 

Health measures (%) 

Health insurance coverage 

Public health insurance 

70.5 

70.6 

69.5 

64.5 

Employer health insurance 

20.6 

21.5 

24.6 

30.1 

Other health insurance 

3.0 

2.7 

2.7 

1.6 

Not covered 

5.9 

5.2 

3.3 

3.8 

Had annual medical checkup when not sick 

Within the past year 

82.1 

82.1 

82.0 

85.5 

1-2 years ago 

14.3 

14.6 

14.3 

11.8 

More than 2 years ago 

3.4 

3.1 

3.4 

2.7 

Never 

0.2 

0.2 

0.0 

0.0 

Last medical checkup was at own (regular) doctor's 

office or clinic (%) 

93.5 

94.5 

96.1 

95.1 

Had preventive dental checkup 

Within the past year 

64.9 

64.8 

66.5 

61.5 

1-2 years ago 

23.5 

23.9 

21.9 

30.5 

More than 2 years ago 

10.9 

10.6 

10.6 

7.5 

Never 

0.7 

0.7 

1.0 

0.0 

Self-rated health 

Excellent or very good 

43.0 

44.3 

39.5 

42.2 

Good 

36.9 

35.5 

36.2 

39.6 

Fair or poor 

20.1 

20.2 

24.3 

18.2 

Over the past two weeks, had little or no interest 

in doing things 

22.4 

22.6 

25.7 

22.5 

Over the past two weeks, had been feeling down, 

depressed, or hopeless 

21.9 

21.9 

23.8 

22.2 
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Table E.l (continued) 


Characteristic 

Full 

Sample 

Core Survey 
Respondent 
Sample 

Child Study 
Sample 

Not Selected 
for Child 
Survey 

Community District (%) 





Brooklyn CD 5 

19.3 

20.8 

20.0 

20.3 

Brooklyn CD 1 6 

15.6 

16.6 

18.0 

19.8 

Bronx CD 5 

15.9 

13.5 

12.4 

11.8 

Bronx CD 6 

17.8 

15.4 

14.7 

18.7 

Manhattan CD 10 

11.7 

12.4 

13.4 

9.6 

Manhattan CD 1 1 

19.7 

21.3 

21.5 

19.8 

Sample size 

4,749 

3,082 

716 

187 


SOURCES: MDRC calculations using data from Baseline Information Forms, New York State unemployment 
insurance (UI) wage records, and New York State Human Resources Administration administrative records. 

NOTES: This table excludes information for enrolled second parents in two-parent households. 

Sample sizes may vary because of missing values. 

Rounding may cause slight discrepancies in calculating sums. 

“Refers to both U.S. citizens by birth and by naturalization. 
b Refers to 30 hours a week or more. 


Comparison of Impacts in the Survey Sample and Report Sample 

A comparison of impacts on educational outcomes from the 1 8-month survey respond- 
ent sample reveals that the pattern of impacts found in the core sample is similar in the report 
sample, though slightly larger in magnitude. As shown in Tables E.3 and E.4, impacts from the 
core study are largely replicated in this subsample of 5 1 1 teenagers. Relatively limited impacts 
on educational outcomes are found for the full sample of teens, as shown in Table E.3, with 
effects on attendance and credits earned, but little by way of statistically significant benefits of 
the program on Regents test scores. Table E.4 presents education effects by eighth-grade math 
proficiency subgroups. When teenagers are split into two groups based on their eighth- grade 
test scores, positive impacts are observed among those teenagers who had scored at or above the 
basic proficiency level on their eighth-grade standardized math test, and these benefits are 
observed in both samples. More specifically, in the 18-month survey sample, Family Rewards 
resulted in a 15 percentage point increase in the likelihood of having a 95 percent or better 
attendance rate (Year 2), a 9 percentage point increase in the likelihood of earning at least 11 
credits in Year 1 (needed to remain on track for on-time graduation), and an increase of 6 
percentage points in the likelihood of passing at least two Regents exams. These numbers are 
slightly larger (at 16, 12, and 12 percentage points, respectively) in the report sample. Overall, 
the patterns of impacts are very similar in the core study and the report samples across sub- 
groups. 
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The Opportunity NYC Demonstration: Family Rewards 

Table E.2 

Characteristics of Teenager Sample at the time of Random Assignment, 
by Full, Core Survey Respondent, Child Study, and Not Selected Samples 


Characteristic 

Full 

Sample 

Core Survey 
Respondent 
Sample 

Child Study 
Sample 

Not Selected 
for Child 
Survey 

Gender (%) 

Female 

49.4 

49.3 

52.3 

48.4 

Male 

50.6 

50.7 

47.7 

51.6 

Age (%) 

0-5 years 

0.1 

0.1 

0.1 

0.0 

6-10 years 

32.2 

32.6 

0.8 

1.1 

11-13 years 

37.2 

39.5 

23.3 

22.5 

14 years or older 

30.5 

27.9 

75.7 

76.5 

Average age (years) 

12 

12 

14 

14 

Born in the U.S. (%) 

92.8 

93.8 

91.9 

93.0 

Race/ethnicity (%) 

Hispanic/Latino 

47.0 

44.1 

42.7 

42.2 

White, non-FIispanic/Latino 

0.4 

0.5 

0.6 

0.5 

Black, non-FIispanic/Latino 

50.5 

53.8 

54.5 

56.7 

Other, non-FIispanic/Latino 

2.0 

1.6 

2.1 

0.5 

Type of school teenager attended in the past year (%) 

Public or charter school 

98.5 

98.6 

99.3 

99.5 

Private or parochial school 

1.5 

1.4 

0.7 

0.0 

Education measures (%) 

Teenager's parent attended parent-teacher 
conference during past year 

Never 

3.6 

3.7 

3.3 

4.9 

1 -2 times a year 

35.8 

35.3 

36.9 

35.3 

3-4 times a year 

36.4 

37.0 

32.4 

37.0 

5-6 times a year 

11.7 

11.1 

12.6 

12.5 

More than 6 times a year 

12.4 

12.9 

14.8 

10.3 

Teenager's parent spoke to teacher about his or her 
grades, tests, or homework during past year 

Not at all 

1.7 

1.5 

1.7 

2.7 

A little 

7.1 

6.9 

6.5 

5.9 

Some 

18.7 

17.1 

18.0 

14.6 

A lot 

36.9 

36.9 

35.9 

43.2 

A great deal 

35.6 

37.6 

38.0 

33.5 

Enrolled in special education in the past 

15.8 

16.1 

13.0 

17.1 

school year 

Enrolled as an English Language Learner 

in the past school year 

13.8 

12.9 

11.5 

9.7 
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Table E.2 (continued) 


Characteristic 

Full 

Sample 

Core Survey 
Respondent 
Sample 

Child Study 
Sample 

Not Selected 
for Child 
Survey 

Teenagers' health outcomes (%) 

Received Medicaid in the 3 quarters prior to 

random assignment 

65.8 

66.0 

66.1 

57.2 

Health insurance coverage 

Public health insurance 

79.1 

78.6 

76.4 

70.4 

Employer health insurance 

16.1 

16.9 

19.3 

24.2 

Other health insurance 

1.7 

1.5 

1.7 

1.6 

Not covered 

3.1 

2.9 

2.5 

3.8 

Parent's rating of teenager's health 

Excellent 

43.0 

44.5 

42.2 

43.3 

Very good 

31.1 

30.4 

30.4 

27.3 

Good 

22.5 

21.7 

24.4 

25.1 

Fair 

3.0 

3.1 

2.4 

3.7 

Poor 

0.3 

0.4 

0.6 

0.5 

Had annual medical checkup when not sick 

Within the past year 

89.5 

89.5 

89.2 

87.2 

1 -2 years ago 

9.5 

9.5 

9.6 

9.6 

More than 2 years ago 

0.9 

1.0 

1.3 

3.2 

Never 

0.1 

0.0 

0.0 

0.0 

Last annual checkup was at own (regular) doctor's 

office or clinic 

97.5 

97.8 

97.8 

98.9 

Had preventive dental checkup: 

Within the past year 

76.5 

75.8 

73.6 

71.2 

1 -2 years ago 

19.2 

19.8 

20.1 

23.9 

More than 2 years ago 

3.3 

3.3 

5.3 

3.3 

Never 

1.0 

1.0 

1.0 

1.6 

Has physical problem that limits his or her 

activities 

10.1 

10.7 

10.7 

9.7 

Has an emotional or mental health problem 

that limits activities 

6.4 

6.4 

5.4 

8.1 

Sample size 

4,749 

3,082 

716 

187 


SOURCES: MDRC calculations using data from Baseline Information Forms, New York State 
unemployment insurance (UI) wage records, and New York State Fiuman Resources Administration 
administrative records. 

NOTES: Sample sizes may vary because of missing values. 

Rounding may cause slight discrepancies in calculating sums. 
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Table E.3 


Impacts on Selected Education Outcomes, Child Study Sample 


Outcome 

Core Sample 

Difference 

(Impact) 

Child Study Sample 

Difference 

(Impact) 

Education outcomes 

Attendance (%) 

Attendance is 95% or higher, Y ear 1 

2.5 

3.4 

Attendance is 95% or higher, Y ear 2 

5 2 *** 

7 7 ** 

Credits (%) 

Attempted 1 1 or more credits, Y ear 1 

3 g *** 

5.7 ** 

Attempted 1 1 or more credits, Y ear 2 

2.6 

5.2 * 

Earned 1 1 or more credits, Year 1 

-0.3 

4.1 

Earned 1 1 or more credits, Year 2 

- 0.2 

3.9 

Regents exams, years 1 and 2 combined 
Exams taken (%) 

At least 1 Regents exam 

2.0 

7 7 ** 

1 Regents exam 

-0.3 

1.0 

2 Regents exams 

- 1.2 

1.9 

3 Regents exams 

3.4 * 

4.7 

Exams passed (%) 

At least 1 Regents exam 

1.6 

5.2 

At least 2 Regents exams 

0.7 

5.3 

1 Regents exam 

0.9 

- 0.1 

2 Regents exams 

- 1.0 

-2.3 

3 Regents exams 

1.7 

7.5 ** 

Average number of Regents exams passed 

0.0 

0.2 

Sample size 

1,979 

511 


SOURCES: MDRC calculations from responses to the Family Rewards 18-Month Survey and 
Family Rewards 30-Month Embedded Study Survey. 

NOTES: Statistical significance levels indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 
percent. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre-random 
assignment characteristics of family or sample members. 

Rounding may cause slight discrepancies in calculating sums and differences. 
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Table E.4 


Impacts on Selected Education Outcomes, Child Study Sample, 
by Proficiency Level on 8th-Grade Math Test 



Core Sample 

Child Study Sample 


Difference 

Difference 

Subgroup and outcome 

(Impact) 

(Impact) 

Scored at or above uroficiencv level 
on annual math test in 8th urade 



Attendance (%) 



Attendance is 95% or higher, Year 1 

7.4 * 

3.1 

Attendance is 95% or higher, Year 2 

14 9 *** 

16.2 ** 

Credits (%) 



Attempted 11 or more credits, Year 1 

4 1** 

g 0 *** 

Attempted 1 1 or more credits, Year 2 

2.6 

10.0 ** 

Earned 1 1 or more credits, Year 1 

g 9 ** 

11.7 * 

Earned 1 1 or more credits, Year 2 

9 2** 

14.6 ** 

Regents exams passed 



At least 1 Regents exam, Year 1 (%) 

7.5 ** 

11.6 * 

At least 1 Regents exam, Year 2 (%) 

5.8 * 

13.6 ** 

At least 2 Regents exams, Year 1 to 2 (%) 

5.9 * 

11.7 * 

Average number of Regents exams passed, Year 1 to 2 

0.3 ** 

0.5 ** 

Sample size 

584 

199 

Scored below proficiencv level 
on annual math test in 8th urade 



Attendance (%) 



Attendance is 95% or higher, Year 1 

2.2 

4.5 

Attendance is 95% or higher, Year 2 

2.5 

3.6 

Credits (%) 



Attempted 1 1 or more credits, Year 1 

5.6 *** 

3.2 

Attempted 1 1 or more credits, Year 2 

2.2 

2.0 

Earned 1 1 or more credits, Year 1 

-3.5 

- 0.1 

Earned 1 1 or more credits, Year 2 

-3.8 

- 2.8 

Regents exams passed 



At least 1 Regents exam, Year 1 (%) 

0.7 

4.0 

At least 1 Regents exam, Year 2 (%) 

- 1.2 

0.4 

At least 2 Regents exams, Year 1 to 2 (%) 

-2.3 

3.3 

Average number of Regents exams passed, Year 1 to 2 

- 0.1 

0.1 

Sample size 

1,143 

312 


(continued) 
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Table E.4 (continued) 

SOURCES: MDRC calculations from responses to the Family Rewards 18-Month Survey and Family Rewards 
30-Month Embedded Study Survey. 

NOTES: Statistical significance levels indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre -random assignment 
characteristics of family or sample members. 

Rounding may cause slight discrepancies in calculating sums and differences. 

Sample sizes may vary because of missing values. 


Because of the slightly larger impacts found among subgroup of teenagers who were 
proficient in eighth-grade math, additional comparisons between the teenagers whose parents 
were in the 1 8-month survey sample and the teenagers in the child study sample by proficiency 
subgroups were conducted. (As described in the sample selection section earlier, these samples 
overlap such that every teenager in the child study sample has a parent who responded to the 
18-month survey). Table E.5 shows that teenagers who met proficiency standards in eighth 
grade were more likely to be female (49 percent versus 45 percent), more likely to be bom in 
the United States (by 5 percentage points), and more likely to have parents who attended parent- 
teacher conferences and spoke to teachers more frequently about their teenager’s schoolwork 
compared with the proficient ninth-grade children of parents in the 1 8-month survey sample. 
Proficient teenagers in the child study sample were more likely to be enrolled in special educa- 
tion in the past year (about 21 percent versus 12 percent) and more likely to be enrolled as 
English language learners (approximately 16 percent versus 1 1 percent) than proficient teenag- 
ers whose parents were in the 18-month survey sample. They were also more likely to have had 
an annual medical check-up in the past year when not sick (87 percent versus 79 percent) and 
less likely to have had a preventive dental check-up (67 percent versus 81 percent). Within the 
group of teenagers who did not meet proficiency standards in eighth grade, there were very few 
differences in baseline characteristics between the two survey samples. 


Characteristics of Respondents and Nonrespondents Within the 
Fielded Sample 

Overall, 77 percent of parents who were contacted gave consent for their children to participate; 
of these, 93 percent of children consented to participate and completed a survey. Among the 
716 parent-teenager dyads who were chosen to be surveyed, 511 dyads completed a survey 
interview, for a response rate of 71 percent. The response rate was 73 percent for the program 
group and 70 percent for the control group. 

Appendix Table E.6 presents selected baseline characteristics for survey respondents 
and nonrespondents. Some differences are to be expected, given that individuals who respond to 
surveys tend to be different, usually less disadvantaged, from those who do not. The table 
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Table E.5 


Characteristics of the Fielded Survey Sample at the Time of Random Assignment, by 
Response Status by Full and Child Study Samples, by Proficiency Level on 8th-Grade Math Test 



Scored at or above proficiency level 
on annual math test in 8th grade 

Scored below proficiency level 
on annual math test in 8th grade 

Characteristic 

Core Survey 
Respondent 
Sample 

Child Study 
Sample 

Core Survey 
Respondent 
Sample 

Child Study 
Sample 

Gender (%) 

Female 

44.9 

49.3 

41.6 

38.2 

Male 

55.1 

50.7 

58.4 

61.8 

Average age (years) 

14 

14 

14 

14 

Born in the U.S. (%) 

88.6 

94.0 

92.8 

92.3 

Race/ethnicity (%) 

Hispanic/Latino 

47.6 

48.9 

42.9 

39.2 

White, non-Hispanic/Latino 

0.0 

0.7 

0.5 

0.3 

Black, non-Hispanic/Latino 

49.5 

48.1 

54.9 

58.2 

Type of school child attended 
in the past year (%) 

Public or charter school 

100.0 

100.0 

99.0 

98.5 

Private or parochial school 

0.0 

0.0 

1.0 

1.5 

Education measures (%) 

Teenager's parent attended parent-teacher 
conference during past year 

Never 

6.9 

3.0 

2.6 

2.6 

1 -2 times a year 

41.2 

32.8 

35.0 

36.8 

3-4 times a year 

31.4 

29.9 

32.7 

31.8 

5-6 times a year 

11.8 

14.9 

13.8 

13.6 

More than 6 times a year 

8.8 

19.4 

15.8 

15.2 

Teenager's parent spoke to teacher about 
his or her grades, tests, or homework 
during past year (%) 

Not at all 

4.8 

1.5 

1.0 

0.3 

A little 

12.5 

5.2 

6.7 

8.4 

Some 

20.2 

17.0 

16.3 

16.1 

A lot 

35.6 

37.0 

38.3 

36.7 

A great deal 

26.9 

39.3 

37.7 

38.6 

Enrolled in special education in the past 

school year 

12.4 

20.6 

19.8 

19.2 

Enrolled as an English Language Learner 

in the past school year 

10.8 

15.7 

13.3 

12.6 


(continued) 
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Table E.5 (continued) 



Scored at or above proficiency level 
on annual math test in 8th grade 

Scored below proficiency level 
on annual math test in 8th grade 

Characteristic 

Core Survey 
Respondent 
Sample 

Child Study 
Sample 

Core Survey 
Respondent 
Sample 

Child Study 
Sample 

Teenagers' health outcomes (%) 

Received Medicaid in the 3 quarters prior to 

random assignment 

52.2 

59.4 

61.1 

60.8 

Health insurance coverage 

Public health insurance 

65.2 

65.2 

72.2 

75.1 

Employer health insurance 

31.9 

30.4 

24.3 

21.8 

Other health insurance 

1.4 

1.4 

1.6 

1.0 

Not covered 

1.4 

2.9 

1.9 

2.0 

Parent's rating of teenager's health 

Excellent 

46.4 

50.7 

42.5 

39.9 

Very good 

31.9 

26.1 

30.8 

31.3 

Good 

21.7 

21.7 

25.7 

27.8 

Fair 

0.0 

0.0 

0.6 

1.0 

Poor 

0.0 

1.4 

0.3 

0.0 

Had annual medical checkup when not sick 

Within the past year 

79.4 

87.0 

91.1 

92.4 

1 -2 years ago 

19.1 

11.6 

8.6 

7.6 

More than 2 years ago 

1.5 

1.4 

0.3 

0.0 

Never 

0.0 

0.0 

0.0 

0.0 

Last annual checkup was at own (regular) 

doctor's office or clinic 

97.0 

98.5 

98.7 

99.0 

Had preventive dental checkup: 

Within the past year 

81.2 

66.7 

72.9 

74.2 

1 -2 years ago 

11.6 

29.0 

20.4 

17.2 

More than 2 years ago 

4.3 

2.9 

5.7 

7.6 

Never 

2.9 

1.4 

1.0 

1.0 

Has physical problem that limits his or her 

activities 

2.9 

4.3 

8.3 

8.7 

Sample size 

385 

268 

691 

448 


SOURCE: MDRC calculations using Family Rewards Baseline Information Forms data. 

NOTES: Includes sample members randomly assigned between July 10, 2007, and March 18, 2008. 
Rounding may cause slight discrepancies in calculating sums and differences. 

Sample sizes may vary because of missing values. 


illustrates that there are only a few of these differences. A higher fraction of the respondent 
sample, for example, came from a two-parent family. Similarly, a smaller fraction of respond- 
ents than nonrespondents reported good health. On the other hand, survey respondents received 
more food stamp payments than nonrespondents in the year before random assignment. 
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Table E.6 

Characteristics of the Fielded Survey Sample 
at the Time of Random Assignment, by Response Status 


Characteristic 

Survey 

Respondents 

Non- 

Respondents 

Assigned to program group (%) 

51.3 

46.8 

Family baseline measures 

Two-parent family 3 (%) 

18.6 

12.7 * 

Two parents enrolled in Family Rewards b (%) 

5.7 

2.0 ** 

Number of children in household (%) 

1 

23.7 

25.4 

2 

36.6 

36.6 

3 

39.7 

38.0 

Primary language spoken at home is English (%) 

81.4 

81.5 

Family living in public housing (%) 

31.4 

30.5 

Family receiving Section 8 rental assistance (%) 

23.1 

24.5 

Family receiving TANF or Safety Net Assistance (%) 

22.5 

22.3 

Family receiving food stamps (%) 

59.7 

54.5 

Earnings above 130% of federal poverty level (%) 

14.5 

19.9 * 

UI earnings in the year prior to random assignment ($) 

11,326 

11,582 

TANF payments in the year prior to random assignment ($) 

2,200 

2,059 

Food stamp payments in the year prior to random 

assignment ($) 

2264 

1953 * 

Community District (%) 

Bronx 5 

12.5 

12.2 

Bronx 6 

14.9 

13.2 

Brooklyn 5 

18.8 

23.4 

Brooklyn 16 

18.2 

16.6 

Manhattan 10 

13.9 

13.2 

Manhattan 1 1 

21.7 

21.5 

Parents' baseline measures 

Female (%) 

94.3 

93.7 

Age (%) 

18-34 

15.3 

20.5 

35-44 

50.7 

47.8 

45-54 

27.0 

25.4 

55 or older 

7.0 

6.3 

Citizenship (%) 

U.S. citizen by birth 

70.1 

69.8 

Naturalized U.S. citizen 

15.1 

12.7 

Fegal permanent resident 

14.9 

17.6 
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Table E.6 (continued) 


Characteristic 

Survey 

Respondents 

Non- 

Respondents 

Race/ethnicity (%) 
Hispanic/Latino 

42.7 

45.3 

Black, non-Hispanic/Latino 

55.7 

53.7 

Other, non-Hispanic/Latino 

1.6 

1.0 

Education (%) 

Less than high school diploma/GED certificate 

39.7 

37.1 

High school diploma or GED certificate 

23.2 

24.4 

More than high school diploma/GED certificate 

37.1 

38.6 

Currently working (%) 

49.3 

51.7 

Working full time (%) 

36.6 

39.5 

Covered by public health insurance (%) 

68.1 

71.3 

Medicaid coverage in the 3 quarters prior to random 
assignment (%) 

65.0 

65.9 

Self-rated health is good, very good, or excellent (%) 

73.8 

79.4 

Has a physical or mental condition that limits work (%) 

28.9 

26.9 

Over the past 2 weeks, 

Had little or no interest in doing things and/or 
had been feeling down, depressed, or hopeless (%) 

35.6 

30.4 

Target teenagers' baseline measures 

Bom in the United States (%) 

91.0 

94.1 

Attended public school in past year (%) 

98.8 

100.0 [ ] 

Covered by public health insurance (%) 

76.7 

73.9 

Has a physical or mental condition that limits work (%) 

15.5 

11.9 

Percentage of target teenagers proficient on ELA test, 2007 

35.4 

29.1 

Percentage of target teenagers proficient on math test, 2007 

38.9 

33.7 

Sample size 

511 

205 


SOURCE: MDRC calculations using Family Rewards Baseline Information Forms data. 


NOTES: Includes sample members randomly assigned between July 10, 2007, and March 18, 2008. 

In order to assess differences in characteristics across research groups, chi-square tests were used for 
categorical variables and t-tests were used for continuous variables. 

Significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Brackets ([ ]) around significance levels indicate that the chi-square tests for statistical significance are 
not valid because sample sizes within categories are too small. 

Sample sizes may vary because of missing values. 

“Families with parents who reported their marital status as single, single but living with a boyfriend or 
girlfriend, separated, divorced, or widowed are considered single-parent families; those with parents who 
reported their marital status as married or legal domestic partnership are considered two-parent families. 

b This refers to sample members who enrolled in Family Rewards with their spouse or legal domestic 
partner. 







Differences between respondents and nonrespondents were tested in a regression mod- 
el, in which the probability of response was regressed on a range of baseline covariates. The 
results are shown in Appendix Table E.9. None of the statistically significant differences shown 
in Table E.6 are statistically significant in the regression model. In addition, the full model is 
not statistically significant. Finally, the top row of Table E.6 shows that parent-teenager dyads 
in the control group were not more or less likely to respond to the survey than those in the 
program group. 


Comparison Between the Research Groups in the Survey 
Respondent Sample 

Selected baseline characteristics for program and control group survey respondents are shown 
in Tables E.7 and E.8. The two groups are similar across most dimensions, with a few excep- 
tions. For example, parents in the program group are on average two years older than parents in 
the control group (43 versus 4 1 years old), and parents in the program group are less likely to be 
single. 


Teenagers in the program group also have very similar baseline characteristics to those 
in the control group. The two groups did not differ significantly on any demographic, education, 
or health characteristic at the time of random assignment, except that a smaller percentage of 
teenagers in the program group had a physical problem that limited their activities than in the 
control group (Table E.8). 

These differences are also estimated in a regression framework, in which the likelihood 
of being in the program group is regressed on a range of baseline characteristics (Table E.9). 
None of the differences found in Table E.7 are statistically significant in the full regression 
model, and the model as a whole is not statistically significant. While the differences discussed 
suggest some caution when interpreting survey impacts, these results are reassuring. Moreover, 
the characteristics for which there are differences between the research groups are included in 
the impact regression models. 


Consistency of Impacts 

The previous sections suggested some caution in interpreting the results from the survey 
for two reasons. First, the results for the survey sample may not be generalizable to the full 
research sample, given the difference between the fielded sample and the nonfielded sample in 
parents’ employment activity before and at baseline, health insurance status, teenagers’ race, 
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The Opportunity NYC Demonstration: Family Rewards 
Table E.7 


Characteristics of Adult Sample at the Time of Random Assignment, 

by Research Group 


Characteristic 

Program 

Group 

Control 

Group 

Total 

Female (%) 

94.1 

94.4 

94.3 

Age (%) 

1 8 to 24 years 

0.0 

0.0 

0.0 

25 to 34 years 

14.5 

19.6 

17.0 ** 

35 to 44 years 

46.9 

52.8 

49.9 ** 

45 to 59 years 

34.9 

24.6 

29.7 ** 

60 years or more 

3.6 

3.1 

3.4 ** 

Average age (years) 

43 

41 

42 ** 

U.S. citizen 3 (%) 

83.2 

84.9 

84.1 

By birth 

70.7 

69.6 

70.1 

By naturalization 

12.6 

15.4 

14.0 

Legal permanent resident (%) 

16.8 

15.1 

15.9 

Race/ethnicity (%) 

Hispanic/Latino 

44.4 

42.7 

43.5 [ ] 

White, non-Hispanic/Latino 

0.6 

0.3 

0.4 [ ] 

Black, non-Hispanic/Latino 

54.5 

55.6 

55.1 [ ] 

Other, non-Hispanic/Latino 

0.6 

1.1 

0.8 


Education (%) 

GED certificate 
High school diploma 
Associate's degree/2 -year college 
4-year college or beyond 
None of the above 

Marital status (%) 


Single 

61.5 

69.3 

65.4 

Cohabitating 

2.2 

1.4 

1.8 

Separated, widowed, or divorced 

18.7 

12.8 

15.8 

Married or in a legal domestic partnership 

17.6 

16.5 

17.0 

Primary language is English (%) 

80.4 

82.4 

81.4 

Primary language is Spanish (%) 

19.0 

16.8 

17.9 

Has an account at bank or credit union (%) 

49.3 

44.6 

47.0 

EmDlovment measures 




Currently working (%) 

50.0 

50.0 

50.0 

Working full time b (%) 

37.4 

38.3 

37.8 

Average weekly earnings, among those currently working ($) 

425 

402 

413 

During past year, average number of months worked (%) 

10 

10 

10 


8.9 

11.8 

10.4 

18.9 

20.7 

19.8 

8.5 

7.1 

7.8 

13.3 

10.7 

12.0 

50.4 

49.6 

50.0 
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Table E.7 (continued) 


Characteristic 

Program 

Group 

Control 

Group 

Total 

Employment measures tUI) 

Employed in 4th quarter prior to random assignment (%) 

47.8 

47.2 

47.5 

Average earnings in 4th quarter prior to random assignment ($) 

2,821 

2,474 

2,647 

Employed in 3rd quarter prior to random assignment (%) 

52.0 

46.6 

49.3 

Average earnings in 3rd quarter prior to random assignment ($) 

3,100 

2,753 

2,927 

Employed in 2nd quarter prior to random assignment (%) 

51.1 

43.6 

47.3 ** 

Average earnings in 2nd quarter prior to random assignment ($) 

2,861 

2,625 

2,743 

Employed in 1st quarter prior to random assignment (%) 

48.0 

45.5 

46.8 

Average earnings in 1 st quarter prior to random assignment ($) 

2,947 

2,774 

2,861 

Public assistance measures 1%) 

Received TANF in the year prior to random assignment 

40.2 

39.9 

40.1 

Received Medicaid in the 3 quarters prior to random 

assignment 

62.3 

69.0 

65.6 * 

Health measures 1%) 

Health insurance coverage 

Public health insurance 

68.0 

70.9 

69.5 

Employer health insurance 

26.3 

22.8 

24.6 

Other health insurance 

2.8 

2.6 

2.7 

Not covered 

2.8 

3.7 

3.3 

Had annual medical checkup when not sick 

Within the past year 

81.5 

82.5 

82.0 [ ] 

1 -2 years ago 

14.6 

14.1 

14.3 [ ] 

More than 2 years ago 

3.4 

3.4 

3.4 [ ] 

Never 

0.0 

0.0 

0.0 [ ] 

Last medical checkup was at own (regular) doctor's 

office or clinic (%) 

96.5 

95.7 

96.1 

Had preventive dental checkup 

Within the past year 

67.3 

65.6 

66.5 [ ] 

1 -2 years ago 

21.1 

22.7 

21.9 [ ] 

More than 2 years ago 

10.4 

10.8 

10.6 [ ] 

Never 

1.1 

0.9 

1.0 [ ] 

Self-rated health 

Excellent or very good 

36.9 

42.0 

39.5 

Good 

37.7 

34.7 

36.2 

Fair or poor 

25.4 

23.2 

24.3 

Over the past two weeks, had little or no interest 

in doing things 

27.2 

24.1 

25.7 

Over the past two weeks, had been feeling down, 

depressed, or hopeless 

26.2 

21.5 

23.8 

Sample size 

358 

358 

716 
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Table E.7 (continued) 

SOURCES: MDRC calculations using data from Baseline Information Forms, New York State 
unemployment insurance (UI) wage records, and New York State Human Resources Administration 
administrative records. 

NOTES: In order to assess differences in characteristics across research groups, chi-square tests were used 
for categorical variables and t-tests were used for continuous variables. 

Significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Sample sizes may vary because of missing values. 

Rounding may cause slight discrepancies in calculating sums. 

This table excludes information for enrolled second parents in two-parent households . 

“Refers to both U.S. citizens by birth and by naturalization. 

b Refers to 30 hours a week or more. 

and special education and ELL status, and given the differences in single-parent status, receipt 
of food stamps, and self-rated health between individuals who responded to the survey and 
those who did not. Second, although accounted for in the impact regression model, there were a 
couple of differences in characteristics between program and control group respondents. 

This section helps to put the survey results in context, by comparing impacts estimated 
from administrative data for the research, eligible, fielded, and respondent samples. Impacts for 
the research sample represent the best estimate of the program’s effects, given that they use the 
full program group and control group, and not a potentially nonrandom subset of survey respond- 
ents. Thus, finding similar impacts for the survey sample and the larger research sample would 
give more credibility to the survey analysis. Table E.10 and Table E.ll present the results, 
showing impacts for employment outcomes, using unemployment insurance (UI) records data, 
and education outcomes, using data from New York City Department of Education records. 

Data for employment outcomes (Table E.10) show a roughly similar story for the four 
samples, although the impacts tend to be a bit more negative for the respondent sample than for 
the research sample. 1 For example, the effect on “ever employed” for the research sample is -2.3 
percentage points and statistically significant, compared with -8 percentage points and statistically 
significant for the respondent sample. Recall that program group survey respondents had higher 
Ul-based earnings in the year before random assignment than did control group respondents, 
although this difference is controlled for in the impact regression model. Nonetheless, the data in 
Tables E.10 and E.ll suggest that program’s impacts on employment and earnings and educa- 
tional outcomes for the respondent sample are generally representative of the larger sample. 

Table E.ll presents impacts on education outcomes for the research, eligible, fielded, 
and survey respondent samples. The overall story is the same across all samples. 


'The employment data for the research sample includes all participating parents in each family, while the 
employment data for the fielded and respondent samples include only the primaiy adult. 
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The Opportunity NYC Demonstration: Family Rewards 
Table E.8 

Characteristics of Teenager Sample at the time of Random Assignment, 

by Research Group 


Characteristic 

Program 

Group 

Control 

Group 

Total 

Gender (%) 

Female 

53.4 

51.3 

52.3 

Male 

46.6 

48.7 

47.7 

Age (%) 

0-5 years 

0.3 

0.0 

0.1 ( ] 

6-10 years 

1.1 

0.6 

0.8 [ ] 

1 1-13 years 

22.9 

23.7 

23.3 [ ] 

14 years or older 

75.7 

75.7 

75.7 [ ] 

Average age (years) 

14 

14 

14 

Born in the U.S. (%) 

90.5 

93.3 

91.9 

Race/ethnicity (%) 

Hispanic/Latino 

44.0 

41.4 

42.7 [ ] 

White, non-Hispanic/Latino 

0.6 

0.6 

0.6 [ ] 

Black, non-Hispanic/Latino 

52.9 

56.1 

54.5 [ ] 

Other, non-Hispanic/Latino 

2.5 

1.7 

2.1 

Type of school child attended in the past year (%) 

Public or charter school 

99.7 

98.9 

99.3 ( ] 

Private or parochial school 

0.3 

1.1 

0.7 

Education measures (%) 

Teenager met 8th-grade math learning standards 

39.1 

38.3 

38.7 

Teenager's parent attended parent-teacher conference 
during past year 

Never 

2.3 

4.3 

3.3 

1 -2 times a year 

39.4 

34.5 

36.9 

3-4 times a year 

32.6 

32.2 

32.4 

5-6 times a year 

12.7 

12.4 

12.6 

More than 6 times a year 

13.0 

16.7 

14.8 

Teenager's parent spoke to teacher about his or her 
grades, tests, or homework during past year 

Not at all 

1.4 

2.0 

1.7 

A little 

6.5 

6.4 

6.5 

Some 

20.2 

15.7 

18.0 

A lot 

36.5 

35.3 

35.9 

A great deal 

35.4 

40.6 

38.0 

Enrolled in special education in the past school year 

13.4 

12.6 

13.0 [ ] 

Enrolled as an English Language Learner 

in the past school year 

11.8 

11.3 

11.5 
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Table E.8 (continued) 


Characteristic 

Program 

Group 

Control 

Group 

Total 

Teenasers' health outcomes (%) 

Received Medicaid in the 3 quarters prior to 

random assignment 

65.4 

66.8 

66.1 

Health insurance coverage 

Public health insurance 

76.0 

76.9 

76.4 

Employer health insurance 

21.2 

17.5 

19.3 

Other health insurance 

1.1 

2.3 

1.7 

Not covered 

1.7 

3.4 

2.5 

Parent's rating of teenager's health 

Excellent 

42.2 

42.3 

42.2 ( ] 

Very good 

29.9 

31.0 

30.4 [ ] 

Good 

26.3 

22.5 

24.4 [ ] 

Fair 

1.1 

3.7 

2.4 [ ] 

Poor 

0.6 

0.6 

0.6 [ ] 

Had annual medical checkup when not sick 

Within the past year 

89.9 

88.5 

89.2 [ ] 

1 -2 years ago 

8.7 

10.4 

9.6 r ] 

More than 2 years ago 

1.4 

1.1 

1.3 ( ] 

Never 

0.0 

0.0 

0.0 

Last annual checkup was at own (regular) doctor's 

office or clinic 

97.7 

98.0 

97.8 

Had preventive dental checkup: 

Within the past year 

74.2 

72.9 

73.6 [ ] 

1 -2 years ago 

20.7 

19.5 

20.1 ( ] 

More than 2 years ago 

4.2 

6.5 

5.3 [ ] 

Never 

0.8 

1.1 

1.0 [ ] 

Has physical problem that limits his or her activities 

8.7 

12.6 

10.7 * 

Has an emotional or mental health problem 

that limits activities 

4.8 

6.0 

5.4 

Sample size 

358 

358 

716 


SOURCES: MDRC calculations using data from Baseline Information Forms, New York State 
unemployment insurance (UI) wage records, and New York State Fiuman Resources Administration 
administrative records. 

NOTES: In order to assess differences in characteristics across research groups, chi-square tests were used 

for categorical variables and t-tests were used for continuous variables. 

Significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Brackets ([ ]) around significance levels indicate that the chi-square tests for statistical significance are 
not valid because sample sizes within categories are too small. 

Sample sizes may vary because of missing values. 

Rounding may cause slight discrepancies in calculating sums. 
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The Opportunity NYC Demonstration: Family Rewards 

Table E.9 


Estimated Regression Coefficients for the Probability of Being a 
Program Group Respondent to the Family Rewards Child Study Survey 


Variable 

Respondent Sample 
Parameter 

Estimate P- Value 

Family baseline measures 



Target teenagers proficient on English Language Arts test, 2007 

0.095 

0.657 

Target teenagers proficient on math test, 2007 

-0.023 

0.915 

Community District Bronx 5 

-0.519 

0.147 

Community District Bronx 6 

-0.393 

0.255 

Community District Brooklyn 5 

-0.101 

0.740 

Community District Manhattan 10 

-0.148 

0.654 

Community District Manhattan 1 1 

-0.499 

0.116 

Number of children in household 

0.027 

0.734 

Primary language spoken at home is English 

-0.379 

0.247 

Two-parent family 

0.142 

0.568 

High school diploma, GED certificate, or above 

0.230 

0.256 

C urrently working 

-0.059 

0.793 

Randomly assigned after September 2008 

-0.168 

0.536 

Black, non-Hispanic/Latino 

0.532 

0.490 

Flispanic/Latino 

0.761 

0.329 

U.S. citizen by birth 

0.140 

0.567 

Age 

0.021 

0.110 

Family living in public housing or receiving Section 8 

-0.217 

0.255 

Family receiving TANF or Safety Net Assistance 

0.037 

0.877 

Covered by public health insurance 

-0.009 

0.971 

Likelihood ratio 

19.143 

0.865 

Wald statistic 

16.493 

0.943 

Sample size 

511 



SOURCES: MDRC calculations using Family Rewards Baseline Information Forms and New York 
City Department of Education administrative records. 

NOTES: Estimates were regression-adjusted using ordinary least squares, controlling for pre-random 
assignment characteristics of family or sample members. 

The p-value indicates the likelihood that the difference between the program and control group 
arose by chance. 
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The Opportunity NYC Demonstration: Family Rewards 
Table E.10 


Year 1 Impacts on Ul-Covered Employment and Earnings 
for the Research, Fielded, and Respondent Samples 


Outcome 

Program Group 
Average N 

Control Group 
Average N 

Difference 

(Impact) 

P- Value 

Ever employed (%) 







Research sample 

56.2 

2,513 

58.5 

2,481 

_2 3 ** 

0.011 

Eligible sample 

55.7 

453 

59.5 

450 

-3.8 * 

0.059 

Fielded sample 

58.1 

358 

53.4 

358 

-4.7 ** 

0.043 

Respondent sample 

59.1 

249 

51.1 

262 

-8 0 *** 

0.004 

Average quarterly employment (%) 







Research sample 

49.0 

2,513 

50.3 

2,481 

-1.3 * 

0.091 

Eligible sample 

50.7 

453 

52.3 

450 

- 1.6 

0.340 

Fielded sample 

50.7 

358 

47.8 

358 

-2.9 

0.129 

Respondent sample 

51.2 

249 

46.5 

262 

-4.7 ** 

0.038 

Employed 4 consecutive quarters (%) 







Research sample 

40.8 

2,513 

41.8 

2,481 

- 1.0 

0.252 

Eligible sample 

45.2 

453 

45.6 

450 

-0.4 

0.857 

Fielded sample 

44.2 

358 

41.6 

358 

- 2.6 

0.242 

Respondent sample 

44.4 

249 

40.6 

262 

-3.8 

0.157 

Total earnings ($) 







Research sample 

12,114 

2,513 

12,354 

2,481 

-240 

0.284 

Eligible sample 

13,506 

453 

13,704 

450 

-198 

0.677 

Fielded sample 

12,879 

358 

12,910 

358 

31 

0.950 

Respondent sample 

12,597 

249 

12,565 

262 

-32 

0.956 


SOURCES: MDRC calculations using data from Baseline Information Forms, New York State 
unemployment insurance (UI) wage records, and New York State Human Resources Administration 
administrative records. 

NOTES: Estimates were regression-adjusted using ordinary least squares, controlling for pre-random 
assignment characteristics of family or sample members 

Significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Sample sizes may vary because of missing values. 

The p-value indicates the likelihood that the difference between the program and control group arose by 
chance. 
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The Opportunity NYC Demonstration: Family Rewards 
Table E.ll 


Impacts on Attendance and Credits 
for the Research, Fielded, and Respondent Samples 


Outcome 

Program Group 
Average N 

Control Group 
Average N 

Difference 

(Impact) 

P-Value 

9th-graders at random assignment (%) 

Attendance is 95% or higher, Year 1 

Research sample 

34.0 

988 

31.6 

991 

2.5 

0.226 

Eligible sample 

39.4 

453 

33.0 

450 

6.4 ** 

0.036 

Fielded sample 

37.8 

358 

33.7 

358 

4.0 

0.237 

Respondent sample 

39.8 

262 

36.4 

249 

3.4 

0.422 

Attendance is 95% or higher, Year 2 

Research sample 

28.8 

988 

23.7 

991 

5 2 *** 

0.006 

Eligible sample 

32.5 

453 

25.1 

450 

y 4 *** 

0.009 

Fielded sample 

31.3 

358 

24.8 

358 

6.5 ** 

0.037 

Respondent sample 

34.1 

262 

26.4 

249 

77 ** 

0.048 

Attempted 1 1 or more credits, Year 1 

Research sample 

87.8 

988 

83.9 

991 

2 9 *** 

0.006 

Eligible sample 

91.8 

453 

84.9 

450 

g # 9 *** 

0.001 

Fielded sample 

93.5 

358 

87.8 

358 

5 8 *** 

0.003 

Respondent sample 

93.8 

262 

88.1 

249 

5 7** 

0.013 

Attempted 1 1 or more credits, Year 2 

Research sample 

80.5 

988 

77.9 

991 

2.6 

0.126 

Eligible sample 

84.7 

453 

80.3 

450 

4.4 * 

0.074 

Fielded sample 

85.2 

358 

81.5 

358 

3.7 

0.158 

Respondent sample 

90.4 

262 

85.2 

249 

5.2 * 

0.056 


SOURCES: MDRC calculations using data from Baseline Information Forms, New York State 
unemployment insurance (UI) wage records, New York State Human Resources Administration administrative 
records, and New York City Department of Education Records. 

NOTES: Estimates were regression-adjusted using ordinary least squares, controlling for pre-random 
assignment characteristics of family or sample members 

Significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Sample sizes may vary because of missing values. 

The p- value indicates the likelihood that the difference between the program and control group arose by 
chance. 
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Conclusion 


Overall, the variety of tests that were conducted and the results that are presented suggest that 
the survey sample provides valid estimates of the program’s effects and that these effects are 
representative of those that would have been obtained for the full research sample. Although the 
survey sample differed from the full sample in terms of U.S. citizenship and residency status 
and teenagers’ health insurance status, the administrative records impacts for the survey sample 
are similar to those for the full research sample. 


124 



References 


Adato, Michelle, and Lucy Basset. 2008. What is the Potential of Cash Transfers to Strengthen 
Families Affected by HIV and AIDS? A Review of the Evidence on Impacts and Key Policy 
Debates. Washington. DC: International Food Policy Research Institute. 

Bedsworth, William, Susan J. Colby, and Joe Doctor. 2006. Reclaiming the American Dream. 
Boston: The Bridgespan Group. 

Behrman, Jeremy R., Susan W. Parker, and Petra E. Todd. 2005. Long-Term Impacts of the 
Oportunidades Conditional Cash Transfer Program on Rural Youth in Mexico. Discussion 
Paper 122. Gottingen, Germany: Ibero-America Institute for Economic Research. 

Behrman, J., P. Sengupta, and P. Todd. 2000. The Impact of PROGRESA on Achievement Test 
Scores in the First Year. Washington, DC: International Food Policy Research Institute. 

Blashfield, R. K. 1976. “Mixture Model Tests of Cluster Analysis: Accuracy of Four Agglom- 
erative Hierarchical Methods.” Psychological Bulletin 83, 3: 377-388. 

Borgen, F. H., and D. C. Barnett. 1987. “Applying Cluster Analysis in Counseling Psychology 
Research.” .Journal of Counseling Psychology’ 34, 4: 465-468. 

Coie, J. D., and G. Krehbiel. 1984. “Effects of Academic Tutoring on the Social Status of Low- 
Achieving, Socially Rejected Children.” Child Development 55: 1465-1478. 

Costello, E. J., S. N. Compton, G. Keeler, and A. Angold. 2003. “Relationships Between 
Poverty and Psychopathology: A Natural Experiment.” Journal of the American Medical 
Association 290: 2023-2029. 

Deci, E. L. 1971. “Effects of Externally Mediated Rewards on Intrinsic Motivation.” Journal of 
Personality and Social Psychology > 18: 105-115. 

Deci, E. L., and R. M. Ryan. 2000. “Self-Determination Theory and the Facilitation of Intrinsic 
Motivation, Social Development, and Well-Being.” American Psychologist 55: 68-78. 

Eccles, Jacquelynn, and Allan Wigfield. 2002. “Motivational Beliefs, Values, and Goals.” 
Annual Review of Psychology’ 53: 109-132. 

Fiszbein, Ariel, and Norbert Schady. 2009. Conditional Cash Transfers: Reducing Present and 
Future Poverty. Washington, DC: The World Ba nk . 

Furrer, C., and E. Skinner. 2003. “Sense of Relatedness as a Factor in Children’s Academic 
Engagement and Performance.” .Journal of Educational Psychology’ 95: 148-162. 

Gershoff, E. T., J L. Aber, and C. C. Raver. 2005. Child Poverty in the United States: An 
Evidence- Based Conceptual Framework for Programs and Policies. Pages 269-324 in R. 
M. Lemer, F. Jacobs, and D. Wertlieb (eds.), Applied Developmental Science: An Ad- 
vanced Textbook. Thousand Oaks, CA: Sage. 


125 



Gertler, P. J. 2004. “Do Conditional Cash Transfers Improve Child Health? Evidence from 
PROGRESA’s Control Randomized Experiment.” American Economic Review 94: 336- 
341. 

Greenberg, David, Nadine Dechausay, and Carolyn Fraker. 2011. Learning Together: How 
Families Responded to Education Incentives in New York City ’s Conditional Cash Trans- 
fer Program. New York; MDRC. 

Kao, Grace, and Jennifer Thompson. 2003. “Racial and Ethnic Stratification in Educational 
Achievement and Attainment.” Annual Review of Sociolog y 29: 417-442. 

Kim, Sue, Allen LeBlanc, Pamela Morris, Greg Simon, and Johanna Walter. 2010. Working 
toward Wellness: Telephone Care Management for Medicaid Recipients with Depression, 
Eighteen Months After Random Assignment. New York: MDRC. 

Loeber, R., and T. J. Dishion. 1983. “Early Predictors of Male Delinquency: A Review.” 
Psychological Bulletin 94: 325-382. 

Lugo-Gil, J., and H. Yoshikawa. 2006. Assessing Expenditures on Children in Low-Income , 
Ethnically Diverse and Immigrant Families. Ann Arbor, Ml: National Poverty Center, 
Working Paper Series #06-36. 

Michalopoulos, Charles, Johannes M. Bos, Robert Lalonde, and Nandita Verma. 2000. As- 
sessing the Impact of Welfare Reform on Urban Communities: The Urban Change Project 
and Methodological Considerations. New York: MDRC. 

Midgley, C., M. L. Maehr, L. Z. Hruda, L. Anderman, K. Freeman et al. 2000. Manual for the 
Patterns of Adaptive Learning Scales. Ann Arbor, Ml: University of Michigan. 

Miller Gaubert, Jennifer, Virginia Knox, Desiree Principe Alderson, Christopher Dalton, Kate 
Fletcher, and Meghan McCormick. 2010. Early Lessons from the Implementation of a Re- 
lationship and Marriage Skills Program for Low-Income Married Couples. New York: 
MDRC. 

Milligan, G. W. 1996. Clustering Validation: Results and Implications for Applied Analyses. 
Pages 341-375 in P. Arabie, L. J. Hubert, and G. De Soetes (eds.), Clustering and Classifi- 
cation. River Edge, NJ: World Scientific Publishing. 

Oipinas, P., and R. Frankowski. 2001. “The Aggression Scale: A Self-Report Measure of 
Aggressive Behavior for Y oung Adolescents.” Journal of Early Adolescence 2 1 : 50-67. 

Skoufias, Emmanuel. 2005. PROGRESA and Its Impacts on the Welfare of Rural Households in 
Mexico. Washington, DC: International Food Policy Institute. 

Rawlings, Laura. 2005. “A New Approach to Social Assistance: Latin America’s Experience 
with Conditional Cash Transfer Programmes.” International Social Security Review 58, 2- 
3: 133-161. 

Reynolds, C. R., and R. W. Kamphaus. 1998. Behavioral Assessment System for Children. 
Circle Pines, MN: American Guidance Service Inc. 


126 



Riccio, James, Nadine Dechausay, David Greenberg, Cynthia Miller, Zawadi Rucks, and 
Nandita Verma. 2010. Toward Reduced Poverty Across Generations: Early Findings from 
New York City’s Conditional Cash Transfer Program. New York: MDRC 

Rivera, J. A., J. P. Habicht, T. Shamah, D. Sotres-Alvarez, and S. Villalpando. 2004. “Impact of 
the Mexican Program for Education, Health, and Nutrition (Progresa) on Rates of Growth 
and Anemia in Infants and Young Children: A Randomized Effectiveness Study.” Journal 
of the American Medical Association 291: 2563-70. 

Robin, A. L., and S. L. Foster. 1989. Negotiating Parent- Adolescent Conflict: A Behavioral- 
Family Systems Approach. New York: Guilford Press. 

Roeser R. W., J. S. Eccles, and C. Freedman-Doan. 1999. “Academic Functioning and Mental 
Health in Adolescence: Patterns, Progressions, and Routes from Childhood.” Journal of 
Adolescent Research 14,2: 135-174. 

Roeser, R. W., J. S. Eccles, and K. R. Strobel. 1998. “Linking the Study of Schooling and 
Mental Health: Selected Issues and Empirical Illustrations at the Level of the Individual.” 
Educational Psychologist 33, 4: 153-176. 

Rouse, Colleen. 2004. “Low-Income Students and College Attendance: An Exploration of 
Income Expectations.” Social Science Quarterly 85, 5: 1299-1317. 

Ryan, R. M., and J. P. Connell. 1989. “Perceived Locus of Causality and Internalization: 
Examining Reasons for Acting in Two Domains.” Journal of Personality and Social Psy- 
chology 57: 749-76 1 . 

Schultz, T. Paul, 2001. “School Subsidies for the Poor,” FCND Discussion Paper 102. Wash- 
ington, DC: International Food Policy Research Institute (IFPRI). 

Schultz, T. Paul. 2004. “School Subsidies for the Poor: Evaluating the Mexican Progresa 
Poverty Program.” Journal of Development Economics 74: 199-250. 

Seidman, E., and L. Allen, J. L. Aber, C. Mitchell, and J. Feinman. 1994. “The Impact of 
School Transitions in Early Adolescence on the Self-System and Perceived Social Context 
of Poor Urban Youth.” Child Development 65: 507-522. 

Stattin, H., and M. Kerr. 2000. “Parental Monitoring: A Reinterpretation.” Child Development 
71,4:1072-1085. 


127 




Earlier Publications on Opportunity NYC-Family 
Rewards 


Learning Together: How Families Responded to Education Incentives in New York City ’s 

Conditional Cash Transfer Program 

2011. David Greenberg, Nadine Dechausay, Carolyn Fraker 

Toward Reduced Poverty Across Generations: Early Findings from New York City ’s Conditional 
Cash Transfer Program 

2010. James Riccio, Nadine Dechausay, David Greenberg, Cynthia Miller, Zawadi Rucks, 
Nandita Verma 


NOTE: A complete publications list is available from MDRC and on its Web site 
( www.mdrc.org), from which copies of reports can also be downloaded. 


129 




About MDRC 


MDRC is a nonprofit, nonpartisan social and education policy research organization dedicated 
to learning what works to improve the well-being of low-income people. Through its research 
and the active communication of its findings, MDRC seeks to enhance the effectiveness of so- 
cial and education policies and programs. 

Founded in 1974 and located in New York City and Oakland, California, MDRC is best known 
for mounting rigorous, large-scale, real-world tests of new and existing policies and programs. 
Its projects are a mix of demonstrations (field tests of promising new program approaches) and 
evaluations of ongoing government and community initiatives. MDRC’s staff bring an unusual 
combination of research and organizational experience to their work, providing expertise on the 
latest in qualitative and quantitative methods and on program design, development, implementa- 
tion, and management. MDRC seeks to leam not just whether a program is effective but also 
how and why the program’s effects occur. In addition, it tries to place each project’s findings in 
the broader context of related research — in order to build knowledge about what works across 
the social and education policy fields. MDRC’s findings, lessons, and best practices are proac- 
tively shared with a broad audience in the policy and practitioner community as well as with the 
general public and the media. 

Over the years, MDRC has brought its unique approach to an ever-growing range of policy are- 
as and target populations. Once known primarily for evaluations of state welfare-to-work pro- 
grams, today MDRC is also studying public school refomis, employment programs for ex- 
offenders and people with disabilities, and programs to help low-income students succeed in 
college. MDRC’s projects are organized into five areas: 

• Promoting Family Well-Being and Children’s Development 

• Improving Public Education 

• Raising Academic Achievement and Persistence in College 

• Supporting Low-Wage Workers and Communities 

• Overcoming Barriers to Employment 

Working in almost every state, all of the nation’s largest cities, and Canada and the United 
Kingdom, MDRC conducts its projects in partnership with national, state, and local govern- 
ments, public school systems, community organizations, and numerous private philanthropies. 



